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Abstract 


Fear  of  the  real  or  perceived  consequences  of  receiving  a  bid  protest  exists. 
U.S.  Navy  contracting  officers  have  some  concern  of  protests.  This  concern  can  be 
linked  to  certain  consequences  on  acquisition  strategies.  There  is  enough  qualitative 
and  quantitative  empirical  evidence  to  suggest  that  fear  of  protest  can  impact  what 
would  otherwise  be  prudent  business  decisions.  The  greatest  concerns  are  a  few 
instances  of  inappropriate  uses  of  lowest  price  technically  acceptable  and  the 
reduced  technical  evaluation  effectiveness  attributed  to  fear  of  protests.  If  fear 
waters  down  the  source  selection  hindering  its  ability  to  distinguish  between  the  true 
value  of  offers,  then  contracting  officers  must  ask  themselves  why  go  through  the 
trouble  of  a  best-value  source  selection?  Could  contracting  officers  simply  award  to 
the  low  bidder?  To  what  extent  is  the  set  of  stringent  source  selection  rules  driving 
the  acquisition  team  to  this  result  by  default  (i.e.,  regardless  of  source  selection 
method  actually  employed)?  Thus,  for  the  sake  of  stringent,  fairness-based  rules, 
contracted  outcomes  can  be  compromised.  Whether  the  tradeoff  is  prudent  remains 
to  be  determined.  Further  research  is  needed  to  ascertain  these  other  culprits,  then 
compare  the  relative  effects  of  fear  of  protest  among  other  factors. 

Keywords:  Contracting,  acquisition  strategies,  fear,  protest,  LPTA,  trade  off, 
source  selection,  procurement  administration  lead  time,  competence,  risk, 
requirement  criticality,  discussions,  technical  evaluation  effectiveness,  performance, 
transaction  cost,  authority 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Acknowledgments 


LT  Brian  Colbert,  SC,  USN,  would  like  to  thank  his  wife  Kim  for  her  support 
and  taking  care  of  home  while  he  was  away  at  Navy  Postgraduate  School.  He  would 
also  like  to  thank  his  daughter  Alexis,  for  putting  up  with  her  Dad  being  away  so 
much.  Lastly,  he  would  like  to  thank  Jason  and  Suquon  for  their  work  on  the  project. 

LCDR  Jason  Calandruccio,  SC,  USN,  would  like  to  thank  his  wife,  Heather, 
and  unborn  baby  girl,  Ella,  for  providing  the  motivation  and  inspiration  for  everything 
he  does.  He  would  also  like  to  thank  his  teammates,  Suquon  and  Brian,  for  their 
hard  and  countless  hours  spent  accomplishing  this,  at  times,  overwhelming  task. 

LCDR  Suquon  Combs,  SC,  USN,  would  like  to  thank  first  and  foremost,  God 
for  allowing  him  to  attend  and  complete  his  course  of  study  at  the  Naval 
Postgraduate  School.  He  would  like  to  thank  his  wife,  Zenaida,  for  her  overwhelming 
support  during  his  time  at  the  Naval  Postgraduate  School  and  the  Navy.  Lastly,  he 
would  like  to  thank  Brian  and  Jason  for  their  dedication,  hard  work,  tireless  efforts, 
and  friendship. 

We  are  extremely  grateful  to  the  Deputy,  Assistant  Secretary  of  the  Navy 
(Acquisitions  and  Procurement)  and  staff  for  the  sponsorship  of  this  research.  We 
would  like  to  thank  the  major  contributors  to  our  research  as  well:  The  Navy  Survey 
Manager,  the  Naval  Postgraduate  School  President,  the  Naval  Postgraduate  School 
Institutional  Review  Board  and  staff,  Commander,  Naval  Space  and  Warfare 
Headquarters  and  Commander,  Fleet  Logistics  Center  San  Diego,  Mr.  Daniel 
Gordon  for  his  insights,  and  lastly  those  who  participated  in  the  interviews,  pre¬ 
surveys,  and  surveys. 

Additionally,  we  would  like  to  thank  the  Acquisition  Research  Program,  RADM 
James  Greene,  USN  (ret.),  Ms.  Karey  Shaffer,  and  Ms.  Tera  Yoder,  for  providing  the 
resources  to  ensure  the  success  of  this  project. 

Finally,  we  would  like  to  especially  thank  Lt.  Col  (ret.)  Timothy  Hawkins,  PhD, 
and  Mr.  Elliot  Yoder.  This  research  would  not  have  been  possible  had  it  not  been  for 
their  vision,  tireless  efforts,  guidance,  and  mentorships  throughout  this  project, 
especially  when  we  wanted  to  “abandon  ship”. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


About  the  Authors 


LCDR  Suquon  Combs  is  a  native  of  Augusta,  Ga.  He  was  commissioned  as 
a  Naval  Flight  Officer  after  attending  the  Naval  Academy  Preparatory  School  in 
Newport,  Rl,  and  the  United  States  Naval  Academy  in  Annapolis,  MD,  where  he 
graduated  with  a  Bachelor  of  Economics  degree  in  2003.  In  2004,  he  was  re¬ 
designated  as  a  Supply  Corps  Officer. 

His  shore  assignments  include  the  Customer  Service  Officer,  NAVSUP  Fleet 
Logistics  Center,  San  Diego  from  2007  to  2010.  Half  way  through  his  tour,  he 
participated  in  a  year  long  Individual  Augment  assignment  in  Afghanistan  as  an 
Embedded  Training  Team  Advisor.  As  an  Advisor,  he  mentored  five  Afghanistan 
National  Army  senior  officers  in  logistics. 

His  first  sea  tour  was  the  Food  Service  Officer  and  Sales  Officer  onboard  the 
USS  COWPENS  (CG-63)  in  Yokosuka,  Japan  from  2005  to  2007.  His  second  sea 
tour  was  the  Supply  Officer  onboard  USS  DUBUQUE  (LPD-8)  from  2010  to  201 1 
during  her  final  deployment  and  decommissioning.  Finally,  he  served  as  the  Stock 
Control  Officer  onboard  USS  PELELIU  (LHA-5)  from  2011-2012. 

After  graduation  in  June  2014,  he  will  report  to  the  Naval  Space  and  Warfare 
Center  San  Diego,  CA  as  a  Contracting  Officer. 

Lieutenant  Brian  Colbert  was  born  in  Los  Angeles,  CA.  He  graduated  from 
Gentry  High  School  in  Indianola,  MS  in  1993.  Lieutenant  Colbert  enlisted  in  the  US 
Navy  on  6  February  1996  and  received  training  at  SK  (storekeeper)  “A”  School  with 
follow-on  training  at  Basic  Enlisted  Submarine  School. 

Lieutenant  Colbert’s  first  sea  tour  was  as  a  member  of  the  Storekeeper 
division  onboard  the  USS  HAMPTON  (SSN767).  During  this  tour,  Lieutenant 
Colbert  made  Second  Class  Petty  Officer  and  obtained  several  SK  NECs.  His  next 
tour  was  at  Priority  Material  Office  Norfolk  serving  as  a  Fleet  Expeditor.  During  this 
tour,  Lieutenant  Colbert  obtained  his  Bachelor  of  Science  Degree  in  Computer 
Information  Systems  from  St.  Leo  University.  Afterwards,  Lieutenant  Colbert 
attended  Officer  Candidate  School  in  Pensacola,  FL  and  was  commissioned  on 
March  19,  2004. 

Upon  completion  of  several  months  of  training,  Lieutenant  Colbert’s  served  as 
Supply  Officer  of  the  USS  NEVADA  (SSBN733B)  from  December  2004  to  June 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  V  - 


2007.  His  next  assignment  was  as  a  Naval  Acquisition  Contracting  Officer  at  Naval 
Space  and  Warfare  Center  Headquarters,  San  Diego,  CA  from  July  2007  to  October 
2009.  Lieutenant  Colbert  completed  a  GSA  tour  serving  as  the  Chief  of  the 
Infrastructure  Support  Branch  at  the  Joint  Contracting  Command  Iraq,  RCC-Central 
from  December  2009  to  June  2010.  His  next  assignment  was  to  the  USS  BOXER 
(LHD  4),  where  he  served  as  the  Stock  Control  Officer  and  Principal  Assistant  for 
Logistics  from  August  2010  to  December  2012.  He  is  currently  obtaining  a  Masters 
in  Business  Administration  from  the  Naval  Postgraduate  School. 

He  is  a  201 1  Navy  and  Marine  Association  Leadership  Award  winner  with 
qualifications  as  Submarine  Supply  Corps  Officer,  Naval  Aviation  Supply  Officer,  and 
Surface  Warfare  Supply  Corps  Officer.  His  decorations  also  include  Joint  Service 
Commendation  Medal,  Navy  Commendation  Medal  (four  awards),  and  the  Navy 
Achievement  Medal  (three  awards).  He  is  married  to  Kim  Colbert  of  Silver  Creek, 

MS  and  has  one  daughter,  named  Alexis. 

LCDR  Calandruccio  currently  attends  the  Naval  Postgraduate  School.  He  is 
attaining  an  MBA  in  the  curriculum  of  acquisition  and  contract  management  and  will 
graduate  June  2014.  Jason  is  a  native  of  Newtown,  CT.  He  graduated  with  a  BS  in 
Information  Technology  from  Southern  Polytechnic  State  University. 

LCDR  Calandruccio’s  sea  tours  include  Disbursing  Officer/Food  Service 
Officer  onboard  USS  CLEVELAND  (LPD7)  and  Supply  Officer  USS  CAPE  ST 
GEORGE  (CG71). 

His  Shore  assignments  include  Defense  Contract  Management  Agency, 
located  in  Syracuse,  NY  where  he  served  as  a  Program  Integrator  for  many  large 
ACAT  level  programs  as  well  as  a  contract  administrator.  Also,  Jason  served  a  one 
year  tour  in  various  areas  of  Afghanistan  providing  administrative  contracting 
support. 

Jason  is  a  qualified  Surface  Warfare  Supply  Corps  Officer  and  has  attained  a 
Level  II  DAWIA  certification  in  Contracting.  His  Awards  include  the  Joint  Service 
Commendation  Medal  (2),  Navy  Commendation  Medal  (3),  and  a  Navy  Achievement 
Medal. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


NPS-CM-14-019 


ACQUISITION  RESEARCH  PROGRAM 
SPONSORED  REPORT  SERIES 


What  are  the  Effects  of  Protest  Fear? 

17  June  2014 

LCDR  Jason  Calandruccio,  USN 
LT  Brian  Colbert,  USN 
LCDR  Suquon  Combs,  USN 

Thesis  Advisors:  Dr.  Timothy  Hawkins,  Military  Assistant  Professor 
Elliot  Yoder,  Senior  Lecturer 

Graduate  School  of  Business  &  Public  Policy 

Naval  Postgraduate  School 


Disclaimer:  The  views  represented  in  this  report  are  those  of  the  author  and  do  not  reflect  the  official  policy 
position  of  the  Navy,  the  Department  of  Defense,  or  the  federal  government. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Table  of  Contents 


I.  Introduction . 1 

A.  Background . 1 

B.  Problem  Statement  and  Gap  in  Literature . 4 

C.  Research  Objectives . 6 

D.  Methodology . 6 

E.  Organization  of  the  Study . 7 

F.  Scope . 7 

G.  Managerial  and  Theoretical  Implications . 8 

H.  Conclusion . 9 

II.  Literature  Review . 11 

A.  Introduction . 1 1 

B.  Bid  Protests . 11 

C.  Public  value  framework . 12 

D.  Stakeholder  Theory . 14 

E.  Hypotheses  and  Conceptual  Model . 17 

1 .  Factors  Affecting  Protest  Fear . 17 

a.  Insufficient  Procurement  Administrative  Lead-Time  Planned17 

b.  Contracting  Officer’s  Competence . 17 

c.  Requirement  Criticality/Importance . 18 

d.  Protest  Risk . 18 

2.  Protest  Fear  and  the  Consequences  to  the  Acquisition  Strategy .  1 8 

a.  Ouality  of  Technical  Evaluation  Effectiveness . 18 

b.  Source  Selection  Method  Fit/Appropriateness . 19 

c.  Contractor  Performance . 20 

d.  Buyer  Satisfaction . 22 

e.  PALT  Actual . 23 

f.  Resources . 23 

g.  Transaction  Costs . 23 

h.  Number  of  Awarded  Contracts . 24 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


i.  Contracting  Officer  Authority . 25 

F.  Conclusion . 26 

III.  Methodology . 29 

A.  Introduction . 29 

B.  Interviews . 29 

C.  Population  and  Sample . 30 

D.  Survey  Design  and  Construct  Measurement . 31 

E.  Survey  Pre-Test . 41 

F.  Research  Design . 41 

G.  Conclusion . 43 

IV.  Results . 45 

A.  Introduction . 45 

B.  Data  Collection . 45 

C.  Sample  Demographics . 48 

D.  Assumptions . 49 

1.  MISSING  DATA . 49 

2.  OUTLIERS . 50 

3.  Normality . 51 

4.  Non-Response  Bias . 54 

E.  Sample  Size . 56 

F.  Exploratory  Factor  Analysis . 57 

G.  Hypothesized  Models . 61 

1 .  Antecedents  of  Fear  Results . 62 

2.  Consequences  of  Fear  Results . 65 

H.  Conclusion . 74 

V.  Discussion  and  Recommendations . 77 

A.  Introduction . 77 

B.  Discussion  and  Implications . 78 

1.  Magnitude  of  Fear . 78 

2.  Insufficient  PALT  Planned . 79 

3.  Contracting  Officer’s  Competence . 80 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  X  - 


4.  Requirement  Criticality/Importance . 81 

5.  Protest  Risk . 81 

6.  Technical  Evaluation  Effectiveness . 82 

7.  Source  Selection  Method  Fit/Appropriateness . 83 

8.  Contractor  Performance . 84 

9.  Buyer  Satisfaction . 84 

10.  PALT  Actual . 85 

1 1 .  Resources . 85 

12.  Transaction  Costs . 86 

13.  Number  of  Awarded  Contracts . 86 

14.  Contracting  Officer  Authority . 86 

C.  Answers  to  Research  Questions . 87 

1 .  Research  Question  1 :  Are  contracting  officers  sufficiently 

concerned  about  bid  protests  to  alter  acquisition  strategies? . 87 

2.  Research  Question  2:  If  so,  what  factors  affect  protest  “fear”?  ....  88 

3.  Research  Question  3:  What  are  the  consequences  to  the 

acquisition  strategy  and  contract  outcomes? . 88 

D.  Implications  and  Recommendations  for  Practitioners . 89 

E.  Limitations  of  the  Study . 92 

F.  Areas  for  Further  Research . 93 

1.  Risk  Tolerance . 93 

2.  Previous  Experience  with  a  Protest . 94 

3.  Inappropriate  Source  Selection  Method . 94 

4.  Awarding  on  Initial  Proposals . 95 

G.  Summary . 95 

References . 97 

Appendix  A.  Survey  Instrument . 103 

Appendix  B.  Survey  Invitations . 157 

Appendix  C.  Endorsement  by  DASN  (AP) . 163 

Appendix  D.  Regression  Output  From  SPSS  VER  22 . 165 


p  ^tSUNTIAI’ERSCIfNTlAM 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  xi  - 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


List  of  Figures 

Figure  1 .  Conceptual  Model . 27 

Figure  2.  Left  Side  Hypothesized  Model . 65 

Figure  3.  Right  Side  of  Hypothesized  Model . 72 

Figure  4.  Statistically  Significant  Model . 73 

Figure  5.  FEAR  Construct  Histogram . 79 


PRAtSTANTlAPERSClfNriAM 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


List  of  Tables 


Table  1 .  Bid  Protest  Statistics  for  Fiscal  Years  2009-2013  (from  GAO,  2014,  p. 


4) . 3 

Table  2.  Summary  Table  of  Hypothesis . 26 

Table  3.  Interview  Questions . 29 

Table  4.  Interview  Demographics . 30 

Table  5.  Construct  Measurement  of  Fear  of  Protest . 32 

Table  6.  Construct  Measurement  of  Contracting  Officer  Authority . 32 

Table  7.  Construct  Measurement  of  Insufficient  PALT  Planned . 33 

Table  8.  Construct  Measurement  of  Technical  Evaluation  Effectiveness . 33 

Table  9.  Construct  Measurement  of  Requirement  Criticality/Importance . 34 

Table  10.  Construct  of  Buyer  Satisfaction . 35 

Table  1 1 .  Construct  Measurement  of  Contractor  Performance . 35 

Table  12.  Construct  Measurement  of  Protest  Risk . 36 

Table  13.  Source  Selection  Method  Appropriateness . 37 

Table  14.  Source  Selection  Method  Fit . 38 

Table  15.  Measurement  of  Number  of  Awarded  Contracts . 39 

Table  16.  Construct  of  Transaction  Costs . 40 

Table  17.  Construct  Measurement  of  Contracting  Officer  Competence . 41 

Table  18.  Summary  Statistics . 47 

Table  19.  Education  and  Certification  Demographics . 48 

Table  20.  Experience . 49 

Table  21 .  Contract  Type  and  Item/Service  Procured . 49 

Table  22.  Univariate  Range  of  Standard  Scores . 51 

Table  23.  Normality  Assessment . 53 

Table  24.  Non-Response  Bias  Hypothesis . 54 

Table  25.  Levene’s  Test  Results . 55 

Table  26.  Univariate  F-Tests  for  Non-Response  Bias . 56 

Table  27.  Factor  Loadings . 58 

Table  28.  Construct  Descriptives . 60 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Table  29.  Cronbach’s  Alpha  Scores . 61 

Table  30.  Model  Summary . 63 

Table  31.  ANOVAa . 63 

Table  32.  Coefficients . 64 

Table  33.  Document  Iteration  Descriptive . 69 

Table  34.  Key  Statistics  from  Right-Side  (Consequences)  Regressions . 70 

Table  35.  Summary  of  Hypothesis  Testing . 75 


PRAtSTANTlAPERSClfNriAM 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  xvi  - 


List  of  Acronyms  and  Abbreviations 


AN  OVA 

analysis  of  variance 

APDP 

Acquisition  Professional  Development  Program 

AT&L 

Acquisition,  Technology,  and  Logistics 

BPA 

blanket  purchase  agreement 

CONUS 

continental  United  States 

COR 

contracting  officer  representative 

CRS 

Congressional  Research  Services 

CPAF 

cost  plus  award  fee 

CPFF 

cost  plus  fixed  fee 

CPIF 

cost  plus  incentive  fee 

CSIS 

Center  for  Strategic  and  International  Studies 

DAWIA 

Defense  Acquisition  Workforce  Improvement  Act 

DOD 

Department  of  Defense 

EFA 

exploratory  factor  analysis 

EOFY 

end  of  fiscal  year 

FASA 

Federal  Acquisition  Streamlining  Act 

FAR 

Federal  Acquisition  Regulations 

FFP 

firm  fixed  price 

FPDS-NG 

Federal  Procurement  Data  System-Next  Generation 

FSC 

federal  supply  code 

FY 

fiscal  year 

GAO 

Government  Accountability  Office 

KO 

contracting  officers 

IDIQ 

indefinite  delivery/indefinite  quantity 

LCS 

Littoral  Combat  Ship 

LPTA 

lowest  price  technically  acceptable 

MAC 

multiple  award  contracts 

MAN OVA 

multiple  analysis  of  variance 

PRAtSTANTlAPERSClfNriA^ 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  XVII  - 


OCONUS  outside  continental  United  States 

OUSD  (AT&L)  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology, 
and  Logistics) 


OJT 

on-the-job  training 

PALT 

procurement  administrative  lead-time 

PCO 

procuring  contracting  officer 

PPT 

price  past  performance  trade-off 

PVF 

Public  Value  Framework 

PSC 

product  service  code 

RFP 

request  for  proposals 

SBA 

Small  Business  Administration 

SLDCADA 

Standard  Labor  Data  Collection  and  Distribution  Application 

SPSS 

Statistical  Package  for  the  Social  Sciences 

SSC 

source  selection  council 

SSMA 

source  selection  method  appropriateness 

SSMF 

source  selection  method  fit 

SST 

source  selection  team 

T&M 

time  and  material 

JSHL 

^fRtfSTANTlAFERSCIENTfcu*  H 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  xviii 

NAVAL  POSTGRADUATE  SCHOOL 

I.  INTRODUCTION 


“Men  go  to  far  greater  lengths  to  avoid  what  they  fear  than  to  obtain 
what  they  desire.”  — Dan  Brown,  The  Da  Vinci  Code 

A.  BACKGROUND 

The  U.S.  government  is  the  “world’s  largest  buyer  of  products  and  services,” 
according  to  the  Small  Business  Administration  (SBA)  (2012,  p.  2).  From  pens  and 
pencils  to  the  joint  strike  fighter  and  littoral  combat  ship  (LCS),  over  $500  billion  in 
goods  and  services  has  been  purchased  annually  since  fiscal  year  (FY)  2007. 
However,  starting  in  FY2008,  year-over-year  contract  spending  has  been  declining 
because  of  fiscal  constraints.  Early  estimates  by  the  Government  Accountability 
Office  (GAO)  project  FY2013  spending  to  be  approximately  $460  billion,  which  is 
down  from  $513  billion  spent  in  FY2012. 

In  an  environment  of  fiscal  prudence,  acquisition  professionals  are  charged 
with  the  proper  stewardship  of  taxpayer  funds.  The  Federal  Acquisition  Regulation 
(FAR)  charges  the  members  of  the  acquisition  team  with  exercising  “personal 
initiative  and  sound  business  judgment  in  providing  the  best  value  product  or  service 
to  meet  the  customer’s  needs”  (FAR  1 .102(d)).  The  FAR  also  states,  “An  essential 
consideration  in  every  aspect  of  the  system  is  maintaining  the  public’s  trust.  Not  only 
must  the  system  have  integrity,  but  the  actions  of  each  member  of  the  team  must 
reflect  integrity,  fairness,  and  openness”  (FAR  1.102-2(c)  (1)).  One  mechanism  that 
is  installed  in  the  system  to  ensure  integrity  and  fairness  is  the  protest  system. 

Protests  ensure  “that  entities  doing  business  with  the  government  can  air  their 
complaints  about  governmental  contracting  processes  and  obtain  relief”  (Manuel  & 
Schwartz,  201 1 ,  p.  3).  Without  a  forum  for  businesses  to  air  these  complaints,  they 
could  become  unwilling  to  do  business  with  the  government.  Protests  also  address 
the  FAR  requirements  of  integrity,  fairness,  and  openness  by  providing  for  the 
“accountability  of  procurement  officials  and  government  agencies  by  highlighting  and 
correcting  mistakes  and  misconduct”  (Manuel  &  Schwartz,  2011,  p.  3). 

According  to  FAR  33.101 ,  a  protest  is  “a  written  objection  by  an  interested 
party”  concerning 

A  solicitation  or  other  request  by  an  agency  for  offers  for  a  contract  for 
the  procurement  of  property  or  services. 

The  cancellation  of  the  solicitation  or  other  request. 

An  award  or  proposed  award  of  the  contract. 
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•  A  termination  or  cancellation  of  an  award  of  the  contract,  if  the  written 
objection  contains  an  allegation  that  the  termination  or  cancellation  is 
based  in  whole  or  in  part  on  improprieties  concerning  the  award  of  the 
contract. 

Only  “interested  parties”  can  file  a  protest.  FAR  33.101  defines  an  interested 
party  as  “an  actual  or  prospective  offeror  whose  direct  economic  interest  would  be 
affected  by  the  award  of  a  contract  or  by  the  failure  to  award  a  contract.”  Interested 
parties  can  file  the  protest  at  any  one  of  the  following  venues:  the  agency, 
Government  Accountability  Office  (GAO),  or  the  U.S.  Court  of  Federal  Claims. 

Agency  protests  must  be  received  either  before  “bid  opening  or  the  closing 
date  for  receipt  of  proposals”  for  issues  with  the  solicitation  (FAR  33.103(e)). 
Unsuccessful  offerors  have  10  days  to  file  a  protest  after  the  impropriety  is  known,  or 
they  have  five  days  after  an  unsuccessful  offeror  debrief  is  conducted.  Alternative 
dispute  resolution  should  be  attempted  prior  to  the  protest  filing.  Agencies  should 
make  their  best  effort  to  resolve  these  protests  within  35  days  (FAR  33.101(g)). 

Protests  to  the  GAO  comprise  the  majority  of  all  protests  filed.  The  same  time 
line  for  submission  of  agency  protests  applies  to  protests  to  the  GAO.  The  protestor 
must  provide  a  copy  to  “the  official  and  location  designated  in  the  solicitation  or,  in 
the  absence  of  such  a  designation,  to  the  contracting  officer,  so  it  is  received  no  later 
than  1  day  after  the  protest  is  filed  with  the  GAO”  (FAR  33.104  (a)(1)).  The 
contracting  officer  must  immediately  suspend  performance  on  the  contract  in  dispute 
if  already  awarded  or  terminate  the  contract.  If  the  contract  has  not  been  awarded, 
the  contracting  officer  should  notify  those  offerors  in  the  competitive  range  (FAR 
33.104  (b)(3)). 

An  overall  summary  of  protest  decisions  from  the  GAO  can  be  found  in  Table 
1 .  For  FY2013,  Congress  added  the  requirement  for  the  GAO  to  include  “a  summary 
of  the  most  prevalent  grounds  for  sustaining  protests”  (GAO,  2014).  During  2013, 
the  most  prevalent  reasons  for  sustainment  were 

•  failure  to  follow  the  solicitation  evaluation  criteria, 

•  inadequate  documentation  of  the  record, 

•  unequal  treatment  of  offerors,  and 

•  unreasonable  price  or  cost  evaluation  (GAO,  2014). 
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Table  1.  Bid  Protest  Statistics  for  Fiscal  Years  2009-2013  (from  GAO, 

2014,  p.  4) 


FY  2013 

FY  2012 

FY  2011 

FY  2010 

FY  2009 

Cases  Filed’ 

2420 

(down  2%2) 

2.475 
(up  5%) 

2,353 
(up  2%) 

2.200 
(up  16%) 

1.080 
(up  20%) 

Cases  Closed 

2.538s 

2,405 

2.202 

2.226 

1.020 

Merit  (Sustain  +  Deny) 
Decisions 

500 

570 

417 

441 

315 

Number  of  Sustains 

87 

106 

67 

82 

57 

Sustain  Rate 

17% 

18.6% 

16% 

10% 

18% 

Effectiveness  Rate4 

43% 

42% 

42% 

42% 

45% 

ADR5  (cases  used) 

145 

106 

140 

150 

140 

ADR  Success  Rate8 

86% 

80% 

82% 

80% 

03% 

Hearings7 

3.36% 

(31  cases) 

6.17% 

(56  cases) 

8% 

(46  cases) 

10% 

(61  cases) 

12% 

(65  cases) 

This  was  the  first  time  since  FY2009  that  protests  did  not  increase  year  over 
year,  but  these  numbers  can  be  misleading.  When  measured  against  the  number  of 
contract  actions  awarded,  the  number  of  protests  represents  less  than  one  percent. 
However,  the  percentage  went  up  slightly  for  FY201 3.  In  FY201 1  and  FY201 2, 
protests  represented  .014  percent  of  contracting  actions  awarded  across  the  federal 
government.  In  FY2013,  that  number  rose  to  .018  percent  of  federal  actions.  So, 
while  the  overall  number  of  protests  filed  declined  by  two  percent,  the  number  of 
contract  actions  also  declined  by  23  percent.  According  to  these  numbers,  the 
protest  situation  got  slightly  worse. 

No  agency  can  prevent  a  protest,  but  a  well-prepared  plan  can  minimize 
potential  grounds  for  protest  (Rumbaugh,  2010,  p.  427).  As  the  GAO  has  identified, 
there  are  numerous  reasons  why  an  interested  party  might  choose  to  file  a  protest: 

A  number  of  analysts  have  suggested  that  companies  are  increasingly 
likely  to  file  protests  when  it  is  in  their  business  interest  to  do  so,  even 
when  they  do  not  believe  there  was  an  error  in  the  procurement 
process.  When  agencies  do  not  adequately  debrief  losing  bidders,  the 
losing  companies  may  file  a  protest  to  determine  why  they  lost  the 
competition.  Other  reasons  companies  may  protest,  include  hoping  to 
influence  the  outcome  of  future  competitions  (akin  to  “yelling  at  the 
referee”);  proving  to  shareholders  and  executive  managers  that  they 
are  doing  everything  they  can  to  win  contracts;  or  even  seeking  to  hurt 
the  competition  by  delaying  a  contract  award.  (Schwartz  et  al. ,  2013,  p 
12) 
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Even  though  the  GAO’s  sustainment  rate  is  minute  when  compared  to  the 
number  of  procurements  awarded  throughout  the  federal  government,  the 
Department  of  Defense  (DOD)  acquisition  workforce  thinks  that  it  is  crucial  to  avoid 
protests  (Gordon,  2011).  An  agency’s  best  policy  to  prevent  a  protest  is  to  mitigate 
the  causes  of  the  protest.  This  desire  to  avoid  a  protest  is  the  driving  force  behind 
acquisition  decisions,  internal  and  external  policies,  and  resources  needed  to 
deliberately  or  subconsciously  remove  the  threat  of  a  protest.  Throughout  the 
remainder  of  this  study,  the  desire  or  priority  to  avoid  bid  protests  is  referred  to  as 
the  fear  of  protests. 

B.  PROBLEM  STATEMENT  AND  GAP  IN  LITERATURE 

According  to  three  Congressional  Research  Service  (CRS)  reports  (Manuel  & 
Schwartz,  2011;  Schwartz,  Manuel,  &  Martinez,  2013;  O’Rourke,  2014),  Gordon’s 
publication  (2013),  and  other  publications  (Knauth,  2013;  Maser  et  al.,  2010; 

Kendall,  2012),  the  fear  of  protests  exist.  Fear  exists  because  no  agency  can 
prevent  a  protest,  and  protests  may  result  in  the  following: 

•  issuing  a  stop  work  order  to  suspend  performance, 

•  reevaluating  proposals, 

•  awarding  proposal  preparation  and  protest  filing  cost  to  the  successful 
protester,  or 

•  terminating  the  awarded  contract  and  re-soliciting  the  requirement 
(Rumbaugh,  2010,  p.  415). 

These  protest  results  add  costs,  deplete  resources,  and  create  other 
intangible  effects  (e.g.,  a  diminished  reputation)  to  an  agency.  Based  on  the  protest 
results,  protests  could  be  considered  a  problem  to  an  agency  or  procurement,  even 
though  protests  are  an  avenue  in  ensuring  fairness,  transparency,  and  accountability 
in  the  federal  acquisition  system.  To  mitigate  the  concern  of  protest,  agencies 
change  their  acquisition  strategies  and  outcomes.  There  is  no  data  that  exist  to  show 
the  true  monetary  and  nonmonetary  effects  of  such  mitigation  as  a  result  of  fear. 

Frank  Kendall,  Undersecretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (AT&L),  spoke  at  a  Center  for  Strategic  and  International  Studies  (CSIS) 
conference  in  Washington,  DC.  The  moderator  asked  Mr.  Kendall, 

My  question — my  concern  goes  to  things  that  I  heard  when  I  was  in  the 
White  House,  but  also  what  I  heard  as  part  of  my  participation  on  the 
procurement  round  table,  and  that’s  the  fear  in  the  acquisition  work 
force  on  the  part  of  our  contracting  officers,  contract  specialists  and 
what  that  fear  is  driving  them  to.  (Kendall,  2012) 

In  response,  Kendall  acknowledged  that  fear  exists  within  the  DOD: 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-4- 


The  fear  of  protest  does  exist  out  there.  It  kind  of  permeates  through 
the  system.  You  know,  I  think  at  the  end  of  the  day  we  should  not  be 
paranoid — or  paralyzed,  is  probably  a  better  word — by  fear  of  protest 
or  by  fear  of  litigation.  (Kendall,  2012) 

Kendall,  who  is  the  head  contracting  official  for  the  DOD,  alluded  to  paralysis  by  the 
fear  of  protest.  Evidence  is  presented  in  this  thesis  that  shows  the  causes  and 
effects  of  this  paralysis. 

There  is  evidence  that  Navy  acquisition  professionals  and  senior  leaders 
were  fearful  of  a  protest  for  the  Navy’s  LCS.  According  to  the  Congressional 
Research  Service  report,  it  was  insinuated  that  the  desire  to  avoid  a  protest  situation 
affected  key  decisions: 

What  role,  if  any,  did  a  desire  by  the  Navy  to  avoid  a  potential  contract 
protest  against  the  Navy’s  down  select  decision  play  in  the  Navy’s 
decision  to  propose  the  alternate  dual-award  strategy?  For  example, 
how  concerned,  if  at  all,  was  the  Navy  that  the  announcement  of  an 
LCS  down  select  decision  might  lead  to  a  contract  protest  and 
controversy  somewhat  like  what  has  been  experienced  in  the  Air 
Force’s  KC-X  aerial  refueling  tanker  acquisition  program?  (O’Rourke, 
2014,  p.  68) 

A  December  13,  2010,  press  report  on  the  LCS  program  stated,  “One  high- 
level  Navy  source  recently  said  that  without  the  dual-ship  approach,  ‘there  is  100 
percent  chance  of  a  protest’”  (Cavas,  2010).  As  a  result  of  this  fear,  the  Navy 
changed  its  acquisition  plan  from  a  down  select  to  a  dual-award  strategy. 

While  scholars  and  the  GAO  have  identified  these  deleterious  effects  of  bid 
protests  on  the  government  (Gordon,  2013),  no  research  or  studies  to  date  have 
quantified  them.  In  other  words,  we  do  not  know  the  magnitude  of  fear  of  protests. 
More  specifically,  we  do  not  know  the  following:  the  extent  that  fear  of  a  protest 
affects  acquisition  strategies,  the  monetary  and  non-monetary  costs  of  a  bid  protest, 
and  the  lengths  that  DOD  acquisition  professionals  would  go  to  avoid  a  protest. 
According  to  Gordon,  in  turn,  the  overwhelming  culture  among  DOD  contracting 
officers  is  to  avoid  protests  to  the  maximum  extent  possible.  Contributing  to  the  level 
of  fear  is  the  fact  that  protests  often  impose  litigation  costs,  termination  costs, 
transaction  costs,  and  opportunity  costs  on  the  government;  add  to  the  workload  of  a 
shrinking  acquisition  workforce;  delay  the  time  of  contractor  performance  and 
delivery;  and  bring  shame  to  the  source  selection  team  (SST).  Even  when  a  bid 
protest  is  denied,  it  usually  holds  up  the  protested  acquisition  (Gordon,  2013). 

Gordon  went  on  to  talk  about  the  effects  of  protest  fear:  “Another  concern 
about  the  cost  of  the  protest  system  relates  to  what  might  be  called  its  indirect 
impact.  Fear  of  protests  is  often  given  as  the  explanation  for  contracting  officers’ 
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preference  for  certain  courses  of  action  over  others”  (Gordon,  2013,  p.  36). 

According  to  one  GAO  report,  protest  fear  can  negatively  influence  agency  behavior. 

Fear  of  protest  may  motivate  agency  officials  to  conduct  more  rigorous 
market  research,  hold  a  competition  instead  of  awarding  a  sole-source 
contract,  or  conduct  more  thorough  and  fair  competition.  On  the  other 
hand,  fear  of  a  protest  could  also  prompt  officials  to  try  to  structure  a 
contract  in  a  manner  they  deem  less  likely  to  be  protested,  such  as 
using  lowest  price  technically  acceptable  award  criteria  instead  of  a 
best-value  competition.  (Schwartz  et  al.,  2013,  p.  1 2) 

When  acquisition  professionals  structure  a  solicitation  using  lowest  price 
technically  acceptable  (LPTA)  source  selection  methodology  instead  of  using  trade¬ 
offs,  they  could  be  hurting  the  industry  and  the  customer.  “The  president  of  a  major 
satellite  services  provider  said  the  U.S.  military’s  ‘lowest  price,  technically 
acceptable’  procurement  strategy  is  stifling  innovation  and  ultimately  shortchanging 
warfighters”  (Magnuson,  2014,  p.  1).  This  comment  suggests  that  one  unintended 
consequence  of  bid  protest  is  an  inappropriate  acquisition  strategy,  namely  the 
source  selection  method.  However,  no  quantifiable  data  exists  that  explores  the 
relationship  between  the  fear  of  protest  and  the  appropriateness  of  the  acquisition 
strategy. 

This  research  will  address  the  causes  and  deleterious  effects  of  protest  fear, 
and  examine  the  effects  of  fear  on  the  acquisition  system.  The  research  will  analyze 
the  data  to  determine  whether  there  are  any  significant  correlations  between  the 
cause  and  effects  of  fear,  and  we  attempt  to  quantify  the  magnitude  of  fear  within  the 
U.S.  Navy.  It  will  then  analyze  the  data  to  present  ways  in  which  acquisition  leaders 
across  the  DOD  can  mitigate  this  fear. 

C.  RESEARCH  OBJECTIVES 

The  objective  of  this  research  is  to  address  in  depth  the  true  causes  and 
effects  of  bid  protest  fear.  Specifically,  we  explore  the  following  research  questions: 

•  Are  contracting  officers  sufficiently  concerned  about  bid  protests  to 
alter  acquisition  strategies? 

•  If  so,  what  factors  affect  protest  fear? 

•  What  are  the  consequences  to  the  acquisition  strategy  and  contract 
outcomes? 

D.  METHODOLOGY 

Because  of  the  nature  of  the  research  questions,  it  is  appropriate  to  use  a 
combination  of  qualitative  and  quantitative  methodologies.  Literature  from  several 
governmental,  public,  and  private  studies  and  reports  are  used  to  develop  the 
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conceptual  model,  hypotheses,  interview  questionnaire,  and  survey.  The  existing 
literature  is  used  to  identify  problems  that  have  not  been  addressed — namely  the 
magnitude,  antecedents  to,  and  consequences  of  a  fear  of  protest.  The  existing 
literature  left  an  opportunity  for  further  research  into  an  area  that  would  be 
considered  beneficial  to  the  acquisition  professionals  across  the  federal  government. 

To  ensure  that  this  research  explored  the  most  likely  and  predominant  cause 
and  effect  variables  of  a  fear  of  protest,  several  interviews  were  conducted  at  two 
major  Navy  contracting  activities.  The  goals  of  the  interviews  were  to  validate  the 
conceptual  model,  identify  potentially  key  omitted  variables,  and  validate  the 
measures  of  variables  for  which  there  existed  no  measurement  scales. 

A  survey  was  deployed  online  to  a  population  of  U.S.  Navy  contracting 
officers  and  specialists.  The  data  was  then  used  to  perform  multiple  regression 
analysis  of  causal  relationships  per  the  conceptual  model. 

E.  ORGANIZATION  OF  THE  STUDY 

This  study  is  broken  up  into  several  chapters.  Chapter  II  is  the  literature 
review.  This  chapter  will  discuss  and  provide  an  overview  of  several  applicable 
theories.  In  Chapter  III  discusses  the  methodology  used,  which  includes  the 
research  and  survey  designs,  interviews,  and  construct  measurements.  The  data 
results,  exploratory  factor  analysis,  and  data  regression  models  are  discussed  in 
Chapter  IV.  Finally,  the  findings  will  present  answers  to  the  research  questions,  and 
offer  managerial  implications  and  recommendations  in  Chapter  V. 

F.  SCOPE 

Due  to  time,  resources,  and  administrative  constraints,  this  research  was 
limited  to  U.S.  Navy  personnel.  Furthermore,  this  research  focused  on  both  military 
and  civilian  contracting  officers  who  make  up  the  Navy’s  acquisition  workforce  within 
the  United  States  and  overseas  locations.  This  research  represents  a  wide  variety  of 
goods  and  services  procured  using  FAR  Part  15  source  selection  methods  above 
$150,000,  a  wide  variety  of  supported  units’  mission,  and  a  range  of  contracting 
commands  within  the  U.S.  Navy.  We  only  analyzed  formal  contracting  practices  in 
accordance  with  FAR  Part  15;  contract  action  awards  pursuant  to  FAR  Parts  13  and 
8  were  not  considered  since  the  probability  of  receiving  a  protest  of  simplified  and 
delivery/task  orders  is  lower.  Under  FAR  Part  8,  protests  are  lower  because  the 
government  is  using  mandatory  sources  for  products  and  services.  Under  FAR  Part 
13, 

Unless  the  contract  action  requires  synopsis  pursuant  to  5.101  and  an 
exception  under  5.202  is  not  applicable,  consider  solicitation  of  at  least 
three  sources  to  promote  competition  to  the  maximum  extent 
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practicable.  Whenever  practicable,  request  quotations  or  offers  from 
two  sources  not  included  in  the  previous  solicitation.  (FAR  13.104) 

Under  FAR  Part  15,  there  is  a  greater  chance  for  protest  because  of  the 
effects  of  best  value  (LPTA  and  trade-off)  under  full  and  open  competition. 

G.  MANAGERIAL  AND  THEORETICAL  IMPLICATIONS 

A  study  that  explores  the  causes  and  effects  of  the  fear  of  protest  is  beneficial 
to  DOD  and  federal  government  acquisition  practitioners.  First,  if  the  fear  of  a  bid 
protest  is  significant,  source  selections  may  result  in  the  selection  of  a  sub-optimal 
contractor,  thereby  compromising  performance  and  buyer  satisfaction.  Alternatively, 
agencies  may  award  excessive  contracts  to  avoid  protests.  This  increases 
transaction  costs  and  contradicts  strategic  sourcing  goals.  It  can  also  unnecessarily 
increase  contractors’  bid  and  proposal  costs  as  they  compete  for  individual  task 
orders  under  multiple-award  contracts  (e.g.,  indefinite  delivery,  indefinite  quantity 
[IDIQ]).  Additionally,  such  fear  may  also  lead  SSTs  to  substantially  increase 
transaction  costs  as  extra  measures  are  taken  to  avoid  bid  protests.  These  extra 
measures  also  delay  receipt  of  contractor  performance,  thereby  impeding  the 
mission. 

Second,  the  acquisition  community  can  use  the  results  of  this  study  to 
become  aware  of  the  effects  of  such  fear  and  its  implications.  With  this  awareness, 
the  acquisition  community  can  decide  where  in  the  acquisition  process  to  allocate 
additional  resources  to  reduce  fear  of  protest  and  the  subsequent  consequences. 
This  strategy  allows  the  acquisition  community  to  be  proactive  instead  of  reactive  in 
dealing  with  fear  and  its  consequences. 

Third,  acquisition  leaders  and  planners  within  the  community  can  use  the 
information  to  quantify  and  assess  the  amount  of  man  hours,  money,  and  people 
(i.e.,  resources)  that  are  being  consumed  because  of  protest  fear.  With  this 
information,  acquisition  leaders  and  planners  can  better  align  their  internal/external 
policies,  acquisition  strategies,  and  resources  to  mitigate  the  causes  and  effects 
driving  the  fear  of  protest,  which  ultimately  can  save  money.  In  addition,  acquisition 
professionals  can  understand  and  quantify  the  resources  spent  on  source 
selections.  With  the  understanding  of  the  amount  of  human  capital,  time,  and 
resources,  the  agency  can  determine  which  internal/external  policies  and  acquisition 
strategies  are  not  cost  effective.  Lastly,  the  amount  of  human  capital,  time,  and 
resources  spent  on  bid  protest  mitigation  is  an  opportunity  cost  of  human  capital, 
time,  and  resources  that  could  have  been  devoted  to  other  critical  areas.  This  is 
critical  for  leaders,  because  as  the  DOD  budget  gets  smaller,  human  capital,  time, 
and  resources  will  become  very  precious  commodities  that  agencies  cannot  afford  to 
waste  or  allocate  to  inopportune  activities. 
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Finally,  this  study  recommends  additional  research  into  the  fear  of  protest 
across  different  government  agencies.  Each  agency  has  different  acquisition 
workforce  compositions,  cultures,  acquisition  policies  and  strategies;  the  U.S.  Navy’s 
perspective  can  only  be  used  as  a  point  of  reference.  This  does  not  reflect  the  true 
level  of  fear,  causes,  and  effects  across  the  federal  government.  But  this  study  can 
be  used  as  a  starting  point  for  further  research. 

H.  CONCLUSION 

In  conclusion,  this  study  examines  the  fear  of  a  bid  protest — a  phenomenon 
purportedly  having  a  significant  effect  on  acquisition  decisions.  This  chapter 
discusses  the  problem  statements  and  literature  gaps  along  with  the  methodology 
that  describes  the  approach  to  collecting  data  to  answer  the  research  questions.  The 
following  chapter  discusses  applicable  theories  along  with  the  research  hypotheses 
and  the  conceptual  model. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

The  Public  Value  Framework  (PVF;  Moore,  2000)  and  Stakeholder  Theories 
can  be  applied  to  the  fear  of  bid  protests  from  an  economic  and  managerial 
perspective.  This  chapter  provides  further  explanation  of  the  study’s  conceptual 
model,  research  objectives,  variables,  and  hypotheses.  A  visual  representation  of 
the  model  and  a  summary  of  the  hypotheses  are  provided  at  the  end  of  the  chapter. 

B.  BID  PROTESTS 

There  are  numerous  reasons  why  an  interested  party  might  file  a  bid  protest. 
According  to  a  2013  CRS  report, 

According  to  analysts,  the  most  common  government  errors  cited  in 
protests  are  poorly  written  or  vague  contract  requirements,  failure  to 
follow  the  process  or  criteria  laid  out  in  the  request  for  proposals,  and 
failure  to  adequately  document  their  findings.  Some  analysts  have 
attributed  these  errors  to  an  inexperienced  or  insufficiently  trained 
acquisition  workforce.  (Schwartz  et  al. ,  2013,  p.  11) 

Although  there  are  many  causes  for  protest,  the  most  typical  include 

•  improper  agency  evaluations, 

•  a  lack  of  meaningful  discussions, 

•  defective  solicitations, 

•  improper  exclusion  from  the  competitive  range, 

•  a  lack  of  cost  realism,  and 

•  agency  bias  or  bad  faith  (Rumbaugh,  2010). 

A  particular  business  could  file  a  protest  whether  the  protest  is  frivolous  or 
not. 

The  threat  of  a  protest  can  influence  agency  acquisition  behaviors.  Instead  of 
having  full  and  open  competition,  end  user  customers  and  acquisition  professionals 
could  structure  a  sole  source  contract.  Instead  of  awarding  a  single  contract  (as 
originally  intended  per  the  acquisition  strategy),  multiple  award  contracts  are 
sometimes  awarded  in  order  to  avoid  a  protest.  In  instances  where  a  trade-off 
source  selection  methodology  is  more  appropriate  in  order  to  attain  the  best  value  to 
the  government,  an  LPTA  methodology  is  sometimes  used.  Instead  of  limiting 
proposals  in  the  competitive  range,  non-qualifying  or  non-competitive  proposals  are 
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retained.  Instead  of  conducting  discussions,  agencies  may  award  without 
discussions. 

Protests  come  with  monetary  and  non-monetary  costs.  An  agency  must  incur 
costs  to  prevent  a  potential  bid  protest  (e.g.,  thoroughly  documenting  and 
substantiating  proposal  evaluations  and  trade-off  decisions),  to  defend  against  an 
actual  protest,  and  to  take  corrective  actions.  The  end  users  bear  costs  as  well, 
since  their  requirements  are  delayed  or  jeopardized.  For  example,  building,  fielding, 
and  sustaining  two  varieties  of  LCS  platforms  are  substantially  more  costly  than 
doing  so  for  one  variety.  All  parties  involved  in  a  bid  protest  incur  the  opportunity 
costs  of  their  best  choice.  For  example,  an  agency  allocating  time  and  people  to 
prevent  a  protest  diverts  time  and  people  from  other  critical  requirements.  The  time 
and  peoples’  salaries  or  wages  are  the  opportunity  cost  for  the  critical  requirements. 

C.  PUBLIC  VALUE  FRAMEWORK 

PVF  was  introduced  by  Harvard  professor  Mark  H.  Moore  and  has  been  used 
to  evaluate  and  identify  value  in,  mainly,  the  public  sector.  Value  in  the  public  sector 
is  much  different  than  it  is  in  the  private  sector.  Often  in  the  private  sector,  industry 
uses  shareholder  value  as  a  means  of  evaluating  itself.  The  private  sector,  however, 
is  much  different.  PVF  has  been  utilized  to  “get  public  managers  thinking  about  what 
is  most  valuable  in  the  service  that  they  run  and  to  consider  how  effective 
management  can  make  the  service  the  best  that  it  can  be”  (Coats  &  Passmore, 
2008,  p.  4). 

PVF  for  application  in  the  government  sector  can  be  explained  by  the 
strategic  triangle  (Heymann,  1987;  Moore,  1995).  These  three  elements  are  public 
value,  legitimacy  support,  and  operational  capability.  In  contrast  to  private  sector 
operations,  the  government’s  strategy  does  not  revolve  around  a  specific  bottom 
line,  such  as  shareholder  wealth.  Contracting  professionals  are  often  satisfying 
multiple  entities,  such  as  regulatory  requirements  (e.g.,  the  FAR),  internal 
customers,  the  private  sector,  and  stewardship  of  taxpayer  resources.  According  to 
Moore,  the  first  element,  value, 

directs  managerial  attention  to  the  value  proposition  that  guides  the 
organization.  For  an  enterprise  to  succeed  in  producing  value,  the 
leaders  of  the  enterprise  have  to  have  a  story,  or  an  account,  of  what 
value  or  purposes  that  the  organization  is  pursuing.  They  need  a 
reason  for  the  organization’s  existence,  a  claim  about  the  way  in  which 
the  world  would  be  made  better  through  the  operations  of  the 
enterprise.  (Moore,  2000,  p.  197-198) 

In  essence,  value  in  a  governmental  organization  equates  to  mission. 

Moore  stated  that  legitimacy  and  support 
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directs  managerial  attention  to  the  question  of  where  the  support  for 
pursuing  the  value  will  come  from.  It  is  not  enough  that  an 
entrepreneurial  leader  judges  some  purposes  to  be  valuable.  Others, 
who  provide  the  necessary  financial  resources  and  authorization,  have 
to  agree  with  that  judgment.  In  government,  those  others  include 
citizens,  elected  representatives,  interest  groups,  and  the  media,  which 
has  been  called  the  ‘authorizing  environment’  of  the  organization 
(Heymann,  1987;  Moore,  1995).  (Moore,  2000,  p.  198) 

With  regard  to  the  fear  of  protest,  there  may  be  an  element  of  shame  if  a  source 
selection  is  protested,  particularly  if  there  is  a  notion  that  management  would  not 
support  the  contracting  officers  and  that  the  protest  may  reflect  poorly  on  them.  High 
dollar  contracts,  in  particular,  hold  great  interest  to  our  media  and  our  elected 
officials.  A  protest  would  reflect  negatively  on  the  contracting  official  as  well  as  the 
contracting  office.  With  these  concerns  in  the  back  of  a  contracting  officer’s  mind, 
there  can  be  a  tendency  to  take  measures  in  order  to  avoid  a  protest  that  can  sub- 
optimally  contribute  to  a  source  selection.  For  example,  the  officer  may  rely  too 
heavily  on  LPTA  rather  than  utilizing  a  best  value  approach.  Fear  of  a  protest  is 
understandable.  Time  has  to  be  devoted  to  address  the  requirements  of  a  protest. 
Ultimately,  contracting  offices  then  have  less  time  and  resources  to  devote  to  other 
requirements.  The  needs  of  the  customer  do  not  stop  because  of  a  protest.  This,  in 
turn,  has  an  adverse  effect  on  contracting  officer’s  legitimacy  and  decreases  the 
value  of  the  contracting  element. 

Finally,  operational  capacity  focuses  attention  on  the  question  of 
whether  sufficient  know-how  and  capability  exist  to  achieve  the  desired 
results.  Often,  this  capability  lies  entirely  in  the  organization  that  the 
manager  leads.  However,  sometimes  it  lies  outside  the  organization’s 
boundary,  and  the  organization  has  to  find  ways  to  engage  capacities 
beyond  its  own  to  achieve  the  desired  result  by  creating  partnerships 
of  various  kinds.  (Moore,  2000,  p.  198) 

Experience  and  workload  levels  can  often  be  elements  that  contribute  to  the  concern 
of  a  protest.  If  contracting  officers  have  adequate  experience,  their  likelihood  to  be 
affected  by  this  fear  can  diminish.  They  have  seen  many  source  selections  and 
know  how  to  ensure  they  have  a  contract  file  that  will  stand  the  test  of  a  protest. 
Conversely,  those  with  lesser  experience  and  a  fear  of  protest  from  a  legitimacy  and 
support  perspective  may  take  additional  steps  to  avoid  a  protest  and,  in  turn,  reduce 
the  value  of  the  organization  by  sub-optimally  organizing  the  contracting  strategy.  In 
contracting  there  are  often  very  busy  times,  most  notably  at  the  end  of  the  fiscal  year 
(EOFY).  As  the  organization  may  be  optimally  manned  for  most  of  the  year,  it  is 
likely  they  are  not  for  the  EOFY.  During  this  time,  requirements  are  coming  in  at  a 
much  faster  rate  and  with  less  time  to  produce  results.  Also  during  this  time, 
operational  capacity  can  become  hindered  and  lead  to  an  increased  fear  of  protest. 
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Contracting  professionals  add  value  to  the  country  because  they  address  the 
operational  needs  of  our  military  and,  at  the  same  time,  provide  fairness  and 
address  the  various  public  policy  issues  that  are  required  by  law  and  regulation. 
When  these  align,  customers  receive  what  they  require  at  a  fair  and  reasonable 
price,  and  this  satisfies  the  requirements  of  governing  policies.  Through  this, 
government  contracting  professionals  add  value  to  their  stakeholders.  Contracting 
officers  sometimes  take  steps  throughout  the  acquisition  process  to  avoid  a  protest, 
such  as  minimizing  discussions  or  even  employing  an  LPTA  source  selection 
process  when  a  full  trade-off  method  is  more  appropriate  (Gordon,  2013,  pp.  36-37). 
When  this  occurs,  our  contracting  system  is  not  optimizing  its  value.  When  it  treats  a 
source  selection  with  a  fear  of  protest,  the  agency’s  value  can  be  reduced. 

Contracting  officers  are  also  accountable  to  provide  fairness  to  commercial 
entities  with  which  they  contract  for  goods  and  services  (FAR  Part  1 ).  Often,  though, 
a  fear  of  a  bid  protest  will  result  in  multiple  award  contracts  including  more 
contractors  than  would  have  been  awarded  if  there  was  no  fear  of  a  bid  protest.  In 
multiple  award  contracts  there  is  a  minimum  dollar  value  that  a  contractor  is 
obligated  to  receive.  This  results  in  increased  spending  of  taxpayer  money  that  could 
have  been  more  efficiently  spent  by  awarding  to  fewer,  more  competitive 
contractors.  This  creates  extra  work  for  the  contracting  officer,  duplicates  inventory, 
can  increase  transportation  costs,  and  results  in  non-optimal  use  of  taxpayer  money, 
and  often  upset  contractors  who  never  get  an  award  under  a  multiple  award  contract 
for  which  they  believed  they  were  competitive.  Although  a  reduced  risk  of  a  protest  is 
accomplished,  the  PVF  is  disturbed,  and  there  is  ultimately  less  value  added  by  the 
contracting  process.  What  this  does  not  accomplish  is  a  best  option  for  the  customer 
or  the  taxpayer,  nor  does  it  provide  fairness  to  the  stronger  contractors. 

D.  STAKEHOLDER  THEORY 

The  stakeholder  theory  explains  the  people  and  entities’  interest  in  a 
particular  organization.  A  stakeholder  in  an  organization  is,  by  definition,  any  group 
or  individual  who  can  affect  or  is  affected  by  the  achievement  of  the  organization’s 
objective  (Freeman,  1984).  According  to  a  stakeholder  model  of  a  corporation 
(Freeman,  2001),  stakeholders  for  a  firm  are  owners,  suppliers,  management, 
employees,  customers,  and  the  local  community.  For  this  research,  the  focus  is  on 
the  contracting  offices  as  the  employees  and  the  end-users  as  the  customers. 
Contractors  are  the  suppliers  providing  a  service  and/or  a  product,  and  the 
acquisition  leaders,  senior  contracting  officers,  staff,  and  supervisors  are  the 
management.  Other  stakeholders  in  this  research  are  congress,  legal  representation 
of  the  contracting  office,  and  taxpayers/citizens.  For  the  employees  (contracting 
officers),  their  stake  in  the  firm  (contracting  activity)  could  be  job  security,  high 
wages,  and  job  satisfaction.  In  return,  employees  are  expected  to  follow  the 
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instructions  of  management  most  of  the  time  (Freeman,  2001 ).  This  presents  an 
issue  if  fear  of  a  protest  exists  at  a  higher  principal  level.  Employees  are  expected  to 
change  their  original  acquisition  strategy  to  continue  to  be  stakeholders. 

Suppliers  (contractors)  are  vital  to  the  success  of  the  firm.  In  turn,  the  firm  is  a 
customer  of  the  supplier  (Freeman,  2001 ).  For  the  contracting  activity,  the 
relationship  with  the  contractor  is  vital.  The  contractor  is  a  supplier  of  proposals, 
market  research  information,  innovation,  and  competition.  What  happens  when  a 
supplier  threatens  the  contracting  activity  with  a  protest  or  there  is  a  fear  of  protest 
from  a  supplier?  The  employees,  management,  and  customers  might  change  their 
original  acquisition  strategy  to  keep  the  supplier  as  a  stakeholder. 

Customers  provide  the  lifeblood  of  the  firm  (Freeman,  2001).  Without 
customers,  the  firm  would  not  exist.  All  organizations  realize  and  understand  this. 
Many  organizations  treat  the  customers  as  the  principal  and  themselves  as  an  agent 
in  the  principal-agent  theory.  To  continue  to  get  revenue  and/or  business,  the  agent 
aligns  or  focuses  the  organization’s  actions  on  the  customer’s  needs.  The  agent 
needs  the  customer  as  a  stakeholder  to  continue  to  exist. 

Management  (acquisition  leaders,  senior  contracting  officers,  and  contracting 
staff  and  supervisors)  plays  an  integral  role.  Managers  are  the  only  group  of 
stakeholders  who  enter  into  a  contractual  relationship  with  all  other  stakeholders. 
Managers  are  also  the  only  group  of  stakeholders  with  direct  control  over  the 
decision-making  apparatus  of  the  firm.  Therefore,  it  is  incumbent  upon  managers  to 
make  strategic  decisions  and  allocate  resources  in  the  manner  most  consistent  with 
the  claims  of  the  other  stakeholders  groups  (Hill  &  Jones,  1992).  Management  must 
keep  the  relationships  among  stakeholders  in  balance.  When  these  relationships 
become  imbalanced,  the  survival  of  the  firm  is  in  jeopardy  (Freeman,  2001 ).  These 
imbalanced  relationships  among  stakeholders  and  the  threat  of  a  protest  could  be  a 
factor  of  fear.  For  example,  a  supplier  (contractor)  protests  an  award  because  the 
contracting  activity  did  not  maintain  a  good  stakeholder  relationship  with  the 
supplier.  Since  there  were  no  discussions  or  meaningful  debriefings,  the  supplier 
may  protest  just  to  get  information.  Management  must  devote  people  (employees), 
time,  and  resources  to  deal  with  the  supplier’s  protest  while  still  meeting  the 
customer’s  needs.  Management  (agent)  must  satisfy  the  customer  and  the  supplier 
base,  take  care  of  its  employees,  and  be  good  stewards  of  the  taxpayers’  (principal) 
money. 

Legal  in  many  contracting  activities  is  part  of  the  management  team.  Its  role  is 
to  advise  the  employees  (contracting  officers)  and  the  management  team  of  the 
legal  aspect  of  acquisition  procurement.  Legal  looks  at  the  procurement  from  a 
different  prospective  to  ensure  there  are  no  grounds  for  a  protest  from  prejudice, 
errors  in  the  procurement  process,  violations  of  statutes  or  regulations,  and  other 
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protestable  actions.  As  a  stakeholder,  legal  offers  advice  in  regard  to  the  source 
selection  method.  Based  on  our  interviews,  legal  has  influence  in  the  source 
selection  method. 

Besides  the  employees,  suppliers,  customers,  and  management,  there  are 
other  stakeholders  that  are  an  integral  part  of  the  contracting  process.  They  are  not 
involved  in  the  day-to-day  operation  of  the  acquisition  process. 

Congress  is  an  important  stakeholder  in  the  acquisition  process.  Congress 
exerts  authority  over  the  contracting  process  through  the  use  of  its  constitutional 
powers  (Cibinic,  Nash,  &  Yukins,  2011,  p.  38).  The  primary  means  of  exercising  its 
authority  are  through  the  enactment  of  laws  establishing  new  programs  and  the 
appropriation  of  funds  to  pay  for  these  programs  (Cibinic  et  al.,  201 1 ,  p.  39).  The 
U.S.  Constitution,  Article  1,  Section  9,  Clause  7,  states  that  “no  money  shall  be 
drawn  from  the  Treasury,  but  in  Consequence  of  Appropriations  made  by  law.” 
Congress  also  exercises  control  and  oversight  over  acquisition  activities  through 
requirements  and  limitations  included  in  procurement  statutes,  authorizations  acts, 
appropriations  acts,  and  other  statutes  (Cibinic  et  al.,  201 1 ,  p.  39).  As  a  stakeholder, 
Congress  authorizes  several  administrative  or  judicial  forums  to  hear  protests 
(Manuel  &  Schwartz,  201 1 ,  p.  3).  Those  reasons  are  to  “ensure  that  entities  doing 
business  with  the  government  can  air  their  complaints  about  governmental 
contracting  processes  and  obtain  relief”  and  “enhance  the  accountability  of 
procurement  officials  and  government  agencies  by  highlighting  and  correcting 
mistakes  and  misconduct”  (Manuel  &  Schwartz,  201 1 ,  p.  3). 

Without  a  protest  forum,  other  stakeholders  (i.e.,  suppliers  and  customers) 
would  be  less  willing  to  do  business  with  the  government,  thus  decreasing 
competition,  which  could  potentially  drive  up  the  cost  of  products  and  services 
(Manuel  &  Schwartz,  201 1 ,  p.  3). 

Taxpayers/citizens  represent  a  wide  range  of  individuals.  They  can  be 
considered  as  suppliers  of  funds  through  government  taxation.  As  part  of  the  general 
public,  taxpayers  can  also  be  considered  customers  in  that  their  tax  dollars  are  spent 
on  public  goods  and  services.  For  example,  taxpayers  pay  taxes  to  fund  the  military. 
The  military  protects  and  defends  the  general  public  in  the  event  of  an  invasion. 
Taxpayers  are  employees,  management,  suppliers,  Congress,  legal,  stakeholders 
and  so  forth  in  the  acquisition  process.  As  taxpayers  in  the  acquisition  process, 
stakeholders  have  a  vested  interest  to  see  that  their  precious  funds  were  not 
wasted,  the  procurement  was  awarded  through  the  best  source  selection  method  for 
that  particular  procurement,  and  the  acquisition  process  was  fair  to  the  public.  This 
presents  a  challenge  when  different  stakeholders  have  different  views  about  which 
source  selection  method  should  be  used. 
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This  research  examines  the  stakeholder  theory  costs  and  associated 
problems  that  arise  from  a  fear  of  protest.  Employees  and  management  are  primarily 
concerned  about  a  protest  from  the  supplier.  All  three  stakeholders  are  trying  to 
meet  the  interest  of  the  customer. 

E.  HYPOTHESES  AND  CONCEPTUAL  MODEL 

1.  Factors  Affecting  Protest  Fear 

There  are  numerous  factors  that  affect  a  fear  of  protest.  This  study  primarily 
focuses  on  three  main  factors  contributing  to  the  research  hypotheses:  literature 
reviews,  theory,  and  logic. 

a.  Insufficient  Procurement  Administrative  Lead-Time  Planned 

Insufficient  procurement  administrative  lead-time  (PALT)  is  the 
inadequate  time  allotted  to  accomplish  an  acquisition.  Insufficient  PALT  is  often  the 
result  of  fiscal  appropriation  constraints  that  commonly  occur  toward  the  end  of  the 
fiscal  year.  Expedited  requirements  and  poor  planning  are  also  common  reasons 
that  can  lead  to  insufficient  PALT.  When  sufficient  lead-time  is  not  allocated  to 
properly  define  requirements,  evaluation  criteria,  and  instructions  to  offerors;  train 
the  technical  evaluators;  evaluate  proposals;  document  evaluations  and  trade-offs; 
and  prepare  for  and  brief  decision  makers,  protestable  errors  are  more  likely  to 
occur.  Thus,  sufficient  time  renders  the  acquisition  team  capable  of  successfully 
performing  a  source  selection.  Absent  sufficient  time,  operational  capability  is 
constrained. 

Hypothesis  1:  Insufficient  PALT  has  a  direct  positive  relationship  on  the 

level  of  fear  of  protest. 

b.  Contracting  Officer’s  Competence 

Competence  is  the  contracting  officer’s  contracting  experience, 
educational  background,  and  the  professional  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA)  certification  level  the  contracting  officer  holds.  There  is 
no  substitute  for  experience  and  for  the  level  of  confidence  one  has  in  his  or  her 
ability  as  a  contracting  officer.  The  more  experience  a  contracting  officer  has,  the 
less  concern  of  a  protest  there  should  be  since  the  officer  has  more  experience  in 
techniques  and  practices  to  prevent  bid  protest.  The  contracting  officer  also  has 
more  knowledge  of  how  to  defend  a  protest  if  received,  and  thus,  may  worry  about 
protests  less  because  of  the  officer’s  increased  operational  capacity  to  handle  a 

Hypothesis  2:  The  greater  a  contracting  officer’s  competence  level,  the 
lower  the  level  of  fear  of  protest. 
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c.  Requirement  Criticality/Importance 

How  critical  or  important  the  requirement  is  to  the  overall  mission  of  the 
customer  will  impact  protest  concerns.  If  the  requirement  is  critical  to  the  customer’s 
mission  effectiveness,  there  could  be  a  tendency  to  use  a  procurement  method  that 
does  not  delay  or  jeopardize  the  award  of  the  contract.  This  is  because  the  value  of 
a  government  agency  (e.g.,  military  unit)  is  determined  by  its  mission  effectiveness 
(Moore,  2000).  If  the  delivery  or  performance  of  a  good  or  service  is  delayed,  the 
unit’s  mission  will  likely  be  delayed  to  some  extent,  compromising  the  unit’s  public 
value. 

Hypothesis  3:  The  level  of  requirement  criticality/importance  has  a 

positive  impact  on  the  level  of  fear  of  protest. 

d.  Protest  Risk 

Risk  is  defined  as  the  product  of  the  magnitude  of  consequences  and 
the  probability  of  their  occurrence.  The  risks  involved  with  a  protest  can  be  added 
time  to  complete  the  acquisition,  delays  to  the  mission,  which  can  directly  and 
negatively  impact  the  war  fighter,  additional  effort  and  resources  required  to  resolve 
the  protest,  and  personal  shame  and  embarrassment  experienced  by  the  acquisition 
team.  Protest  may  also  be  perceived  as  bringing  negative  career  repercussions, 
such  as  preventing  promotion  or  a  direct  admonishment.  The  likelihood  of  any  of 
these  negative  consequences  depends  on  the  circumstances  unique  to  each  bid 
protest.  Nonetheless,  if  the  protest  risks  are  high,  the  consequences  could 
jeopardize  the  unit’s  public  value  (e.g.,  its  mission  effectiveness). 

Hypothesis  4:  As  protest  risk  increases,  the  level  of  fear  of  protest 

increases. 

2.  Protest  Fear  and  the  Consequences  to  the  Acquisition  Strategy 

Fear  is  a  natural  reaction  to  a  threat.  As  a  natural  reaction,  fear  enables 
humans  to  remove  the  threat  deliberately  or  subconsciously.  In  this  case,  fear  of 
protest  could  be  considered  a  natural  reaction  to  the  potential  consequences  if  a  bid 
protest  was  received.  Acquisition  professionals  could  deliberately  or  subconsciously 
remove  the  threat  through  their  many  acquisition  strategy  decisions.  These 
decisions  could  be  considered  actions  taken  to  prevent  a  protest. 

There  are  numerous  consequences  to  the  acquisition  strategy  and  contract 
outcomes  as  a  result  of  fear.  This  study  focuses  on  consequences  identified  from 
our  literature  review  and  interviews  with  subject  matter  experts  and  practitioners. 

a.  Quality  of  Technical  Evaluation  Effectiveness 

According  to  FAR  15.304(b),  evaluation  factors  and  significant  sub 
factors  must  (1 )  represent  the  key  areas  of  importance  and  emphasis  to  be 
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considered  in  the  source  selection  decision;  and  (2)  support  meaningful  comparison 
and  discrimination  between  and  among  competing  proposals. 

Agencies  evaluate  the  competitive  proposals  and  assess  their  relative 
qualities  based  only  on  the  factors  and  subfactors  specified  in  the  request  for 
proposal  (RFP)  (Rumbaugh,  2010). 

As  part  of  the  evaluation,  proposals  are  put  in  a  competitive  range.  The 
contracting  officer  determines  which  proposals  are  within  the  competitive  range 
based  on  the  evaluated  price  and  other  evaluation  factors  included  in  the  RFP 
(Rumbaugh,  2010,  p.  338).  A  low-rated  proposal  could  potentially  be  eliminated  from 
the  competitive  range  if  the  proposal  does  not  meet  the  specified  evaluation  factors 
or  subfactors  in  the  RFP. 

If  the  contracting  officer  and  the  source  selection  authority  are 
concerned  with  a  fear  of  protest,  they  could  lower  the  evaluation  factors  and 
subfactors  for  all  of  the  proposals.  By  lowering  the  factors,  the  lower-rated  proposals 
are  not  eliminated  from  the  competition,  which  mitigates  the  chances  of  a  protest. 
The  lower-rated  proposal  might  be  the  winning  proposal. 

Hypothesis  5:  The  level  of  fear  of  protest  has  a  negative  effect  on  the 

technical  evaluation  effectiveness. 

b.  Source  Selection  Method  Fit/Appropriateness 

According  to  FAR  15.101-1,  a  trade-off  process  is  appropriate  when  it 
may  be  in  the  best  interest  of  the  government  to  consider  an  award  to  a  company 
other  than  the  lowest  priced  offeror  or  other  than  the  highest  technically  rated 
offeror.  This  process  gives  the  source  selection  official  very  broad  discretion  when 
the  offeror  with  the  lowest  cost  to  the  government  has  not  been  evaluated  based  on 
the  non-cost  factors  (Cibinic  et  al.,  2011,  p.  678). 

According  to  FAR  15.101-2,  the  LPTA  source  selection  process  is 
appropriate  when  best  value  is  expected  to  result  from  selection  of  the  technically 
acceptable  proposal  with  the  lowest  evaluated  price.  There  are  many  reasons  why  a 
contracting  officer  might  opt  for  the  LPTA.  One  major  benefit  of  this  strategy  is  that 
the  agency  can  greatly  shorten  the  evaluation  process  because  once  the  low  price 
proposal  has  been  found  to  be  technically  acceptable;  there  is  no  need  to  evaluate 
the  acceptability  of  any  other  proposals.  (Cibinic  et  al.,  201 1 ,  p.  680). 

The  source  selection  appropriateness  depends  on  the  procurement 
requirement.  According  to  one  interviewee, 

Again,  because  my  philosophy  has  always  been  what  is  the 
best  way  to  make  sure  the  customer  gets  what  they  want.  Is  it 
going  to  be  something  that  either  yes,  they  can  offer  or  provide 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  19- 


the  product  or  service;  yes  or  no,  so  it  is  a  slam  dunk.  Low  price, 
technically  acceptable — or  do  you  want  to  have  the  flexibility  of 
best  value  where  it  is  something  that  you  want  to  encourage 
innovation  or  design  or  whatever  else? 

According  to  FAR  Part  6.3,  there  are  many  reasons  to  use  a  method 
other  than  full  and  open  competition.  Based  on  the  situation,  a  contracting  officer 
might  use  any  of  those  reasons  with  a  valid  justification  and  approval.  If  a 
contracting  officer  is  concerned  about  a  protest,  he  or  she  could  structure,  with 
justification,  the  requirement  for  a  sole  source  acquisition.  Having  an  approved 
justification  restricts  competition  to  a  sole  source.  This  prevents  other  companies 
from  protesting.  The  use  of  sole  source  works  well  in  situations  where  there  is  an 
emergency,  the  mission  is  in  jeopardy,  and/or  time  is  very  critical. 

A  contracting  officer  may  not  offer  discussions  because  having 
communications  could  potentially  cause  a  protest.  One  of  the  reasons  businesses 
file  protests  is  because  agencies  do  not  adequately  debrief  losing  companies.  Those 
losing  companies  may  file  a  protest  just  to  get  information  on  why  they  lost  the 
competition  (Schwartz  et  al.,  2013). 

A  contracting  officer  could  also  be  afraid  to  talk  to  businesses.  Every 
word  the  contracting  officer  says  could  potentially  be  used  in  the  protest  case. 

During  a  CSIS  conference  with  Kendall,  the  moderator  described  this  as  “the  fear  of 
conducting  discussions  because  contracting  officers  get  it  into  their  head  that  if  they 
conduct  discussions,  they’re  going  to  end  up  with  a  bid  protest,  and  it’s  just  going  to 
take  too  long,  and  they’ll  get  dinged”  (Kendall,  2012). 

According  to  Rumbaugh,  if  the  agency  has  already  received  good 
proposals,  then  going  into  discussions  is  not  likely  to  provide  more  value.  It’s  “not  a 
good  practice  to  conduct  discussions  to  lower  offerors’  prices  when  initial  offers  are 
fair  and  reasonable”  (Rumbaugh,  2010,  p.  326). 

Hypothesis  6:  There  is  a  direct  positive  relationship  between  fear  of 

protest  and  the  Source  Selection  method  fit/  appropriateness. 

c.  Contractor  Performance 

According  to  Gansler,  Lucyshyn,  and  Arendt  (2009),  competition  is 
important  for  the  DOD  because  it  can  aid  in  increasing  economic  efficiency, 
innovation,  quality,  and  performance.  Competition  can  aid  in  economic  efficiency  (for 
both  systems  and  services)  by  forcing  firms  to  keep  prices  as  low  as  possible  in 
hopes  of  winning  a  contract.  Likewise,  increased  innovation  results  from  competitive 
pressures  as  firms  seek  new  solutions  to  DOD  problems  through  research  and 
development  while  trying  to  differentiate  themselves  from  competitors.  Competition 
also  impacts  quality  as  firms  attempt  to  improve  the  quality  of  a  product  over  time, 
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which  has  the  potential  to  drive  lower-quality  firms  out  of  the  market  altogether. 
Finally,  competition  can  also  directly  impact  performance  improvements  for  both 
weapons  systems  and  services.  (Gansler  et  al. ,  2009,  p.  14) 

The  use  of  LPTA  only  requires  the  lowest  price  among  offerors  that 
meet  the  acceptable  technical  requirements.  In  some  instances  the  quality  of 
technical  evaluations  are  degraded  because  of  the  check  in  the  box  mentality  that 
the  LPTA  approach  brings.  Contractors  have  no  incentive  to  improve  on  quality, 
economic  efficiency,  innovation,  and  performance.  As  a  result,  LPTA  contracts  might 
result  in  lower  standards  and  performance.  According  to  an  article  in  National 
Defense  Magazine,  “He  [president  of  a  satellite  company]  pointed  to  one  study  by 
the  research  firm  Market  Connections  in  October  that  said  LPTA  contracts  may 
result  in  standards  and  performance  being  lowered,  and  less  than  optimal  prices  for 
product  development”  (Magnuson,  2014).  These  lower  standards  and  performance 
cost  the  government  in  schedule  delays,  cost  overruns,  poor  contractor 
performance,  rework,  and  ultimately  unsatisfied  customers.  In  cases  where  the 
requirement  requires  innovation  or  complex,  and/or  risk-taking  requirements  on  both 
the  government  and  contractor,  LPTA  might  not  be  the  appropriate  best  value 
method.  For  those  requirements,  the  government  is  looking  for  the  best  overall 
proposal  value.  If  LPTA  was  used  for  those  innovations  or  complex  or  risky 
requirements,  the  contractor  could  be  unable  to  complete  the  task  because  it  truly 
did  not  understand  the  requirement  reflected  in  the  lowest  price  and  minimum 
acceptable  technical  approach.  The  government  could  have  to  terminate  the 
contract  for  default  and  then  re-solicit  and  re-compete  the  requirement,  which  cost 
funds  and  resources.  “When  the  government  applies  the  strategy  [LTPA]  to 
unsuitable  procurements,  both  the  government  and  the  bidders  lose”.  (Lohfeld,  2012 

p.  1) 

The  use  of  sole  source  versus  competition  does  not  incentivize  the 
contractor  to  improve  on  quality,  economic  efficiency,  innovation  and  performance. 
Sole  source  procurements  create  a  monopoly.  The  monopoly  can  set  its  prices  to 
yield  the  maximum  profit.  The  monopoly  has  no  incentives  to  reduce  price  or 
improve  on  the  product’s  or  service’s  capabilities.  The  government  could  potentially 
pay  for  the  highest  price  for  the  least  amount  of  innovation,  performance,  and 
quality.  The  trade-off  to  the  sole  source  versus  competition  is  the  unusual  and 
compelling  situation  in  which  time  is  of  the  essence. 

Contractor  performance  is  the  measure  of  how  well  the  contractor 
performed  the  contract  based  on  the  fit  of  the  source  selection  and  the  quality  of  the 
technical  evaluation.  Specifically,  it  measures  how  well  the  contractor  met  the 
requirements  in  terms  of  schedule,  cost,  and  performance,  as  well  as  the  terms  and 
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conditions  set  forth  in  the  contract.  Contractor  performance  measures  how  well  the 
contractor  met  the  customers’  expectations. 

Hypothesis  7;  There  is  a  positive  correlation  between  the 
fit/appropriateness  of  a  source  selection  and  contractor  performance. 

Hypothesis  8:  There  is  a  positive  relationship  between  technical 
evaluation  effectiveness  and  contractor  performance. 

d.  Buyer  Satisfaction 

Customer  satisfaction  is  important.  According  to  the  stakeholder 
theory,  customers  are  the  lifeblood  of  any  organization  (firm).  Ensuring  their 
satisfaction  is  critical  to  an  organization’s  existence.  The  role  of  management  within 
the  organization  is  to  ensure  that  the  appreciable  measures  (e.g.,  customer  service, 
highest  quality  products  and  services,  and  professionalism)  are  in  place  to  ensure  its 
customers  keep  coming  back.  For  the  contracting  activity,  the  customer  cannot  set 
up  or  bind  the  government  into  contracts.  The  customer  does  not  know  the  intricate 
procurement  system  of  regulations  and  statutes.  It  is  important  for  the  agency  to  act 
on  the  customer’s  behalf.  The  agency  must  award  contracts  to  suppliers 
(contractors)  to  provide  a  product  or  service  to  the  customer.  The  suppliers  chosen 
to  provide  or  perform  the  service  play  an  important  role  in  customer  satisfaction.  The 
agency’s  commitment  to  the  customer  is  displayed  by  responsiveness  and  resolution 
of  customer  concerns,  problems,  and  complaints  (Anderson,  Swaminathan,  & 

Mehta,  2013). 

If  the  contracting  officer  (employee)  has  a  concern  about  the 
customer’s  concerns,  problems,  and  complaints,  the  contracting  officer  could  change 
the  acquisition  plan  to  using  trade-off  or  LPTA.  The  contracting  officer  could  use 
LPTA  instead  of  trade-off  to  keep  a  certain  contractor  in  the  competition  to  satisfy 
the  customer.  If  the  contracting  officer  is  concerned  about  a  protest,  he  or  she  could 
use  LPTA. 

The  use  of  LPTA  could  potentially  cause  degradation  in  the  customer’s 
satisfaction.  As  companies  reduce  their  costs  to  meet  the  terms  and  conditions  of 
the  contract,  they  cut  key  elements  (e.g.,  people,  level  of  service,  material  quality)  of 
their  productions.  Cutting  key  elements  decreases  the  performance  and  increases 
the  cost  and  the  amount  of  time.  This  degradation  causes  lower  buyer  satisfaction, 
as  the  customer  must  add  more  funds  and  time  to  the  procurement  while  losing  out 
on  the  original  performance  parameters  set. 

Hypothesis  9:  There  is  a  direct  positive  relationship  between 
contractor’s  performance  based  on  the  source  selection  method  used 
and  the  buyer’s  satisfaction. 
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e. 


PALT  Actual 


PALT  is  one  metric  that  can  increase  if  fear  causes  members  of  the 
source  selection  team  to  re-evaluate  themselves.  Naturally,  as  the  concern  over  a 
protest  grows,  acquisition  teams  take  added  measures  to  prevent  them.  This  will 
increase  the  iterations  of  documents  that  are  generated  and  will  ultimately  prolong 
the  review  process.  These  added  measures  consume  time  during  the  source 
selection.  Therefore,  it  is  hypothesized  that: 

Hypothesis  10:  There  is  a  direct  positive  relationship  between  fear  of 

protest  and  the  actual  PALT. 

f.  Resources 

The  composition  of  the  SST  is  customized  for  each  acquisition.  The 
structure  of  the  SST  depends  on  the  nature  of  the  acquisition  and  the  agency’s 
regulations  (Rumbaugh,  2010,  p.  52).  The  source  selection  council  (SSC)  is  another 
group  that  is  added  to  the  SST.  The  SSC  function  is  to  be  advisory  and  an 
independent  reviewer  of  high  dollar  or  high  visible  acquisitions.  When  the  SSC  is  not 
formed,  other  government  advisors  (e.g.,  legal)  can  function  and  provide  their 
expertise  (Office  of  the  Under  Secretary  of  Defense  of  Acquisition,  Technology,  and 
Logistics  (OUSD  [AT&L)),  201 1 ,  p.  7) 

The  typical  source  selection  board  includes  no  more  than  three  to  five 
members.  However,  the  more  significant  procurement  is  to  an  agency,  the  larger  the 
board  is  likely  to  be  (Edwards,  2006,  p.  69).  To  avoid  a  protest,  an  agency  might  add 
more  members  to  the  source  selection  boards. 

Hypothesis  11:  There  is  a  direct  positive  relationship  between  fear  of 

protest  and  the  number  of  Human  Resources  assigned  to  the  source 

selection  team. 

g.  Transaction  Costs 

The  DOD  has  experienced  a  significant  increase  in  the  number  of 
competitive  source  selection  decisions  that  are  protested  by  industry.  Protests  are 
extremely  detrimental  to  the  warfighter  and  the  taxpayer.  These  protest  actions 
consume  vast  amounts  of  the  time  of  acquisition,  legal,  and  requirements  team 
members,  and  delay  program  initiation  and  the  delivery  of  capability  (Schwartz  et  al., 
2009,  p.  13). 

The  transaction  cost  focuses  on  the  organization  of  transactions  that 
occur  whenever  a  good  or  service  is  transferred  from  a  provider  to  a  user  across  a 
technologically  separable  interface.  When  transactions  occur  within  an  organization, 
the  transaction  costs  can  include  managing  and  monitoring  personnel  and  procuring 
inputs  and  capital  equipment  (Pint  &  Baldwin,  1997). 
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In  this  research,  the  transaction  costs  reflect  the  monetary  cost  of 
resources  devoted  to  preventing  a  protest.  This  includes  the  different  acquisition 
professions  (contracting  officer,  contracting  specialist,  technical  evaluator,  legal, 
cost/price  analyst,  past  performance  team,  program  manager,  Small  Business 
Association  [SBA]  representative,  and  consultant),  their  labor  rates,  and  the  amount 
of  their  time.  This  transaction  cost  could  be  considered  an  opportunity  cost  of 
resources  not  devoted  to  other  requirements  and  contracts  in  the  queue. 

Hypothesis  12:  There  is  a  direct  positive  relationship  between  fear  of 

protest  and  transaction  costs. 

h.  Number  of  Awarded  Contracts 

Multiple  award  contracts  (MACs)  are  solicitations  that  are  broken  into 
several  contracts  for  different  companies.  This  includes  task  orders  and  indefinite 
delivery/indefinite  quantity  (IDIQ).  A  contracting  officer  might  use  MACs  because 
they  offer  many  advantages.  According  to  the  Office  of  Management  and  Budget, 

In  order  for  agencies  to  take  continuous  advantage  of  the 
benefits  of  competition  after  contract  award,  Federal  Acquisition 
Streamlining  Act  (FASA)  provides  that  agencies  may  make 
multiple  awards  of  task  and  delivery  order  contracts  for  the 
same  or  similar  supplies  or  services  (and  from  the  same 
solicitation)  to  two  or  more  sources.  The  use  of  multiple  award 
contracts  allows  agencies  to  take  continuous  advantage  of  the 
competitive  forces  of  the  commercial  marketplace  which  will 
result  in  lower  prices,  better  quality,  and  reduced  time  from 
requirements  identification  to  award,  and  improved  contractor 
performance  in  satisfying  customer  requirements.  (Office  of 
Federal  Procurement  Policy  [OFPP],  1997) 

One  of  the  other  advantages  of  using  MACs  is  to  reduce  the  likelihood  of  a  protest 
by  awarding  contracts  to  multiple  companies. 

Under  FAR  16.505  10  (i)  (a),  no  protest  under  FAR  33.1  is  authorized 
in  connection  with  the  issuance  or  proposed  issuance  of  an  order  under  a  task-order 
contract  or  delivery-order  contract,  except  for  “a  protest  on  the  ground  that  the  order 
increases  the  scope,  period,  maximum  value  of  the  contract.” 

For  the  Navy’s  LCS  program,  the  Navy  wanted  to  award  two  contracts 
to  two  competing  shipyards.  This  increased  competition,  lowered  the  cost,  and 
prevented  a  protest.  A  December  13,  2010  press  report  on  the  LCS  program  stated, 
“One  high-level  Navy  source  recently  said  that  without  the  dual-ship  approach,  ‘there 
is  100  percent  chance  of  a  protest’”  (O’Rourke,  2014,  p.  68).  As  a  result  of  this  fear, 
the  Navy  changed  its  acquisition  plan  from  a  down  select  to  a  dual-award  strategy. 
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Hypothesis  13:  The  higher  the  level  of  fear  of  protest,  the  greater  of  the 
number  of  awarded  contracts. 

i.  Contracting  Officer  Authority 

In  some  agencies,  legal  employees  are  considered  advisors. 

According  to  Rumbaugh  (2010,  p.  270),  advisors  are  subject  matter  experts  who 
provide  technical  expertise  and  advice  to  the  evaluators.  Advisors  may  read  the 
proposal,  document  their  findings,  and  make  recommendations  to  the  evaluators. 
Advisors  offer  specific  input  in  their  area  of  expertise  and  may  only  be  asked  to 
evaluate  a  limited  subset  of  evaluation  factors. 

As  advisors,  legal  is  part  of  the  acquisition  process.  Legal  is  invited  to 
the  technical  evaluations.  In  some  agencies,  it  is  an  internal  policy  to  have  legal 
review  proposals  before  they  go  forward.  If  there  is  a  protest,  legal  must  defend  the 
agency’s  actions  against  the  GAO  and/or  the  protester’s  legal  team.  Legal  must 
devote  hours  and  resources  away  from  other  requirements.  In  many  cases,  its  focus 
is  on  the  protest.  Legal  understands  the  importance  of  the  protest  and  the  effects  it 
has  on  the  agency  and  the  customer.  Not  only  is  legal’s  time  devoted  to  the  protest, 
but  the  contracting  officers’  and  evaluators’  time  is  also  a  factor  when  justifying  its 
actions  to  the  Government  Affairs  Office,  protester’s  legal  time,  or  the  parent 
agency.  With  a  lot  of  legal  consequences,  legal  is  overly  conservative  of  preventing 
a  protest.  Since  legal  plays  an  important  role  in  the  acquisition  process,  Contracting 
officers  rely  heavily  on  its  opinion  and  recommendations.  This  statement  is 
supported  by  the  feedback  received  during  our  interviews.  According  to  one 
interviewee,  “They  [legal]  are  only  advisory,  but  we  rely  very  heavily  on  their 
opinion.”  Another  interviewee  shared  the  same  concern: 

Part  of  legal’s  opinion  may  sometimes  tend  to  be  narrow¬ 
minded.  Sometimes  they  tend  to  not  consider  the  real  world. 
How  should  I  put  that?  Hmm.  They  might  see  things  their  way 
and  not  really  how  it  works.  We  almost  never  move  forward 
unless  they  [legal]  give  us  their  okay.  It  would  be  very,  very 
hard — very  challenging. 

Contracting  officers  are  the  only  ones  that  can  legally  bind  the 
government  in  a  contract.  They  have  the  final  authority.  Legal,  in  its  advisory  role, 
influences  the  contracting  officer’s  authority  through  its  opinion  and 
recommendations. 

Hypothesis  14:  There  is  a  direct  negative  relationship  between  fear  of 

protest  and  the  contracting  officer’s  authority. 
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F. 


CONCLUSION 


A  total  of  14  relationships  were  hypothesized  based  on  the  relevant  literature. 
A  depiction  of  the  conceptual  model  can  be  found  in  Figure  1 .  A  summary  of  the 
hypothesis  can  be  found  in  Table  2.  Chapter  III  presents  the  research  and  statistical 
methods  that  are  employed,  along  with  the  measurement  scales  that  are  used  to 
measure  the  relationships  in  the  conceptual  model. 


Table  2.  Summary  Table  of  Hypothesis 

Notation 

Hypothesis 

HI 

Insufficient  PALT  has  a  direct  positive  relationship  on  the  level  of  fear  of  protest. 

H2 

The  greater  a  contracting  officer’s  competence  level,  the  lower  the  level  of  fear  of 
protest. 

H3 

The  level  of  requirement  criticality/importance  has  a  positive  impact  on  the  level  of  fear  of 
protest. 

H4 

As  protest  risk  increases,  the  level  of  fear  of  protest  increases. 

H5 

The  level  of  fear  of  protest  has  a  negative  effect  on  the  technical  evaluation 
effectiveness. 

H6 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  Source  Selection 
method  fit/  appropriateness. 

H7 

There  is  a  positive  correlation  between  the  fit/appropriateness  of  a  source  selection  and 
contractor  performance. 

H8 

There  is  a  positive  relationship  between  technical  evaluation  effectiveness  and 
contractor  performance. 

H9 

There  is  a  direct  positive  relationship  between  contractor’s  performance  based  on  the 
source  selection  method  used  and  the  Buyer’s  satisfaction. 

H10 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  actual  PALT. 

H1 1 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  number  of  Human 
Resources  assigned  to  the  source  selection  team. 

H12 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  transaction  costs. 

H13 

The  higher  the  level  of  fear  of  protest,  the  greater  of  the  number  of  awarded  contracts. 

H14 

There  is  a  direct  negative  relationship  between  fear  of  protest  and  the  contracting 
officer’s  authority. 
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III.  METHODOLOGY 


A.  INTRODUCTION 

This  chapter  begins  by  discussing  the  interview  construction  and  rationalizing 
the  sample  by  discussing  the  method  used  to  sample  the  population  of  contracting 
professionals.  Next,  the  survey  design  and  construct  measurements  are  presented. 
Finally,  the  design  and  validation  of  the  conceptual  model  for  analyzing  the 
antecedents  and  consequences  of  protest  fear  along  with  the  statistical  methodology 
for  analyzing  the  relationships  of  the  model  are  discussed. 

B.  INTERVIEWS 

The  next  step  in  the  validation  process  was  to  interview  subject  matter 
experts.  The  purpose  was  to  validate  the  hypothesized  model  and  to  develop  and 
validate  the  measures  of  constructs  that  did  not  have  existing  measurement  scales. 
As  such,  the  interviews  provided  content  validity  and  construct  validity.  Navy 
contracting  officers  at  the  Commander  Naval  Space  and  Warfare  Headquarters  and 
at  the  Navy  Fleet  Logistics  Center  San  Diego  were  chosen  for  interviews  because  of 
(a)  the  convenience  of  travel,  (b)  a  willingness  to  support  the  research,  and  (c)  the 
availability  of  a  wide  variety  of  contract  types  and  contracted  goods  and  services  for 
wide  generalizability  (e.g.,  external  validity).  A  series  of  questions  (see  Table  3)  was 
asked  to  each  participant.  All  responses  were  unrehearsed  and  not  coerced. 

_ Table  3.  Interview  Questions _ 

Interview  question _ 

How  important  to  you  is  avoiding  a  bid  protest? 

Why  is  avoiding  a  bid  protest  important? 

What  are  the  negative  consequences  of  a  bid  protest? 

Are  there  any  positive  outcomes  of  receiving  a  bid  protest? 

Are  some  members  of  the  source  selection  team  more  fixated  on  avoiding  a  bid 
protest  than  others?  Who?  Why? 

If  there  were  no  ability  to  protest,  would  you  have  done  anything  differently  in  the 
past  on  a  source  selection  (e.g.,  acquisition  strategy  elements)? 

Do  you  believe  that  source  selection  teams  alter  acquisition  strategies  in  order  to 
avoid  bid  protests?  What  are  the  outcomes  of  these  alterations? 

What  extraordinary  measures  have  you  observed  or  heard  of  that  source 
selection  teams  have  taken  to  avoid  a  bid  protest? 


1 

2 

3 

4 

5 

6 

7 

8 


A  total  of  18  individuals  were  interviewed  over  two  days  between  the  two 
commands.  Demographics  of  each  respondent  can  be  found  in  Table  4.  Each 
interview  was  recorded  and  transcribed.  The  average  interview  lasted  26  minutes. 
The  interviews  resulted  in  eight  hours  and  three  minutes  of  recordings,  which 
transcribed  into  229  pages. 
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Interviewees  were  given  a  copy  of  the  conceptual  model  during  the  interview 
and  asked  whether  they  agreed  with  the  independent  variables  being  used.  They 
were  also  asked  if  they  would  add  any  or  take  any  away.  One  of  the  respondents,  a 
supervisory  contracting  officer  with  29  years  of  experience,  stated,  “Okay.  This  is 
good.  I  don’t  see  anything  that  I  need  to  add.”  Another  contracting  officer  with  18 
years  of  experience  who  had  recently  dealt  with  a  pre-award  protest  stated,  “I  think 
this  is  a  great  research  that  you  are  doing  because  this  is  a  bigger  and  bigger  issue. 
I  think  you  are  right  on.”  Other  statements  that  validated  the  model  were,  “I  think  I 
like  the  model.  For  the  most  part  it  says  everything.” 


Table  4.  Interview  Demographics 


Rank/Grade 

Gender 

Duty  Title 

Yrs. 

Exp 

# 

Source 

Sel. 

(within  2 
yrs) 

#  Bid 
Protest 
(within  2 
yrs) 

Contract  Types 

GS-14 

F 

Supv.  Contracting  officer 

29 

12 

0 

CPFF,  T&M, 
FFP 

GS-13 

M 

Contracting  officer 

26 

9 

3 

CPFF,  T&M, 
FFP 

GS-13 

M 

Contracting  officer 

18 

6 

1 

Various 

GS-14 

M 

Supv.  Contracting  officer 

36 

1 

0 

Various 

GS-13 

F 

Contracting  officer 

6 

4 

0 

CPFF,  T&M, 
FFP 

GS-14 

F 

Supv.  Contracting  officer 

32 

3 

0 

Various 

GS-12 

M 

Contract  Specialist 

4 

0 

0 

Various 

GS-13 

M 

Contracting  officer 

6 

6 

0 

Various 

GS-12 

M 

Contract  Specialist 

7 

4 

2 

FFP,  CPFF, 
CPAF 

GS  13 

M 

Contracting  officer 

21 

1 

0 

FFP,  Cost 

GS  13 

F 

Contracting  officer 

30 

4 

0 

FFP,  Cost 

GS  13 

M 

Contracting  officer 

22 

5 

0 

CPFF;  CPIF; 
FFP 

GS  13 

F 

Contracting  officer 

15 

5 

1 

CPFF;  CPAF 

GS  13 

M 

Contracting  officer 

26 

20 

1 

FFP;  CPFF 

C.  POPULATION  AND  SAMPLE 

After  the  interview  testimonies  confirmed  the  conceptual  model,  a  survey  to 
collect  the  quantitative  data  was  developed.  The  population  for  this  study  consisted 
of  only  the  continental  United  States  (CONUS)  and  outside  the  continental  United 
States  (OCONUS)  U.S.  Navy  civilian  and  military  contracting.  The  research  was 
limited  to  the  Navy  because  of  time  and  sponsorship  constraints.  The  sample  was 
constrained  to  contracting  officers  who  had  executed  a  FAR  Part  15-based  formal 
source  selection  with  a  dollar  amount  of  greater  than  $150,000.  The  rationale  was  to 
remove  contracting  officers  who  primarily  execute  acquisitions  below  the  simplified 
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acquisition  threshold  and  who  conduct  non-competitive  procurements.  Several 
constraints  existed  for  surveying  this  population.  A  list  of  e-mail  addresses  was 
generated  using  data  extracted  from  the  Federal  Procurement  Data  System-Next 
Generation  (FPDS-NG)  database  to  encompass  all  transactions  that  fit  the  criteria 
previously  stated.  Because  of  the  costs  required  to  mail  a  sufficient  quantity  of 
surveys  to  contracting  personnel  and  the  resources  required  to  manually  input 
responses  into  a  dataset,  a  web-based  survey  was  deployed  to  each  individual  e- 
mail  address  generated  from  FPDS-NG  data. 

The  unit  of  analysis  for  this  survey  was  a  source  selection.  Since  nearly  all  bid 
protests  stem  from  a  protestable  action  (e.g.,  a  proposal  rating,  rating  justification,  or 
basis  of  a  trade-off  analysis)  associated  with  a  source  selection,  this  is  the  proper 
unit  of  analysis  for  the  study.  Respondents  were  instructed  to  answer  the  survey 
questions  using  their  experience  from  their  most  recently  completed  FAR  Part  15 
source  selection.  The  most  recent  source  selection  was  required  to  serve  as  the 
basis  of  reference  in  order  to  prevent  respondents’  self-selection  bias. 

D.  SURVEY  DESIGN  AND  CONSTRUCT  MEASUREMENT 

There  were  115  questions  used  to  measure  the  16  variables  in  the  conceptual 
model.  The  interview  informants  reviewed  the  measures  of  new  constructs  and 
found  their  content  to  capture  the  full  meaning  of  the  constructs  being  measured.  All 
scales  measuring  latent  constructs  used  a  Likert-type  scale.  Questions  collecting 
continuous  data,  such  as  dollar  value,  planned  PALT,  actual  PALT,  iterations  of 
source  selection  documents,  and  quantity  of  personnel  assigned  did  not  use  Likert- 
type  scales. 

Fear  of  protest  is  a  term  used  for  this  research  to  identify  the  level  of 
apprehension  a  contracting  professional  has  about  receiving  a  bid  protest.  Since  the 
fear  of  protest  construct  was  substantiated  during  the  literature  review  and  through 
qualitative  interviews,  six  questions  were  constructed  to  measure  the  level  of  fear  of 
protest  among  the  respondents  to  the  survey,  shown  in  Table  5.  No  previously- 
validated  scales  were  available  to  measure  the  fear  of  protest.  One  variable  from 
this  construct  was  discarded  as  a  result  of  factor  analysis. 
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Table  5.  Construct  Measurement  of  Fear  of  Protest 


Scale  Item 

Survey  Question 

FEAR1* 

At  some  point  during  the  development  of  the  acquisition  strategy  or  the  source 
selection  process,  1  worried  about  receiving  a  bid  protest. 

FEAR"* 

1  was  concerned  that  the  contract  award  would  be  protested. 

FEAR3* 

1  was  anxious  to  get  beyond  the  10-day  point  after  contract  award  (or  debriefings) 
to  determine  whether  or  not  the  contract  would  be  protested. 

FEAR4*# 

Receiving  a  bid  protest  would  have  been  among  the  worst  things  that  could 
happen. 

FEAR3** 

During  the  development  of  the  acquisition  strategy  and  throughout  proposal 
evaluation,  to  what  extent  were  you  concerned  that  an  offeror  might  protest  the 
contract  award? 

FEAR3** 

During  the  development  of  the  acquisition  strategy  and  throughout  proposal 
evaluation,  to  what  extent  was  at  least  one  other  member  of  the  source  selection 
team  concerned  that  an  offeror  might  protest  the  contract  award? 

*  anchors  of  strongly  agree  and  strongly  disagree 
**  anchors  of  not  at  all  concerned  and  extremely  concerned 
#  item  was  discarded  due  to  Exploratory  Factor  Analysis 

The  contracting  officer  authority  construct  described  how  empowered  the 
contracting  officer  is  to  make  final  decisions  during  the  source  selection  process. 
The  contracting  officer  authority  factor  was  validated  through  qualitative  interviews. 
Six  questions  were  constructed  to  measure  the  contracting  officer’s  authority  to 
make  decisions,  which  are  shown  in  table  6.  There  were  no  previously-validated 
scales  available  for  this  construct.  The  construct  consisted  of  six  questions  using  a 
seven-point  Likert-type  scale  with  anchors  of  strongly  agree  and  strongly  disagree. 
One  variable  from  this  construct  was  discarded  as  a  result  of  factor  analysis. 

Table  6.  Construct  Measurement  of  Contracting  Officer  Authority 


Scale  Item 

Survey  Question 

CO_AUTH 1 

1  was  empowered  to  make  required  decisions  throughout  the  source  selection. 

CO_AUTH2 

1  was  trusted  that  the  decisions  1  made  throughout  the  source  selection  would  be 
appropriate. 

CO_AUTH3 

My  management  supported  me  on  the  decisions  1  made  during  the  source  selection. 

CO_AUTH4 

If  1  disagreed  with  an  aspect  of  a  legal  opinion/review,  1  had  the  latitude  to  deviate 
from  it. 

CO_AUTH3#* 

1  had  to  change  documents  generated  during  the  source  selection  to  correspond  with 
reviewers. 

CO  AUTH6* 

1  might  as  well  not  have  a  warrant  since  my  decisions  were  overridden  by  reviewers. 

#  item  was  discarded  due  to  Exploratory  Factor  Analysis 

*  item  was  reverse  coded 


jaBS/g 

V  RAtSWrnAJ|E^CI£N77^^, 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-32- 


The  antecedent  insufficient  PALT  was  validated  by  qualitative  interviews  and 
vetting  by  contracting  practitioners.  Three  questions,  shown  in  Table  7,  were 
constructed  to  measure  the  sense  of  urgency  created  by  insufficient  PALT  planned 
in  the  milestones  and  allocated  by  the  acquisition  team  and  its  managers  to  conduct 
the  source  selection.  The  scales  were  developed  from  a  study  on  services  sourcing 
performance  (Muir,  2010).  The  variable  consisted  of  three  questions  using  a  seven- 
point  Likert-type  scale  with  anchors  of  strongly  agree  and  strongly  disagree. 

_ Table  7.  Construct  Measurement  of  Insufficient  PALT  Planned _ 

Scale  Item _ Survey  Question _ 

PALT_PU  The  milestones  for  awarding  this  contract  were  too  aggressive. 

PALT_P2  I  was  not  rushed  to  award  this  contract. 

PALT_P3  I  had  sufficient  time  to  get  this  contract  awarded. 

*  item  was  reverse  coded 

The  technical  evaluation  effectiveness  construct  was  also  validated  by 
qualitative  interviews.  Six  questions  were  constructed,  shown  in  Table  8,  to  measure 
the  technical  evaluators’  level  of  confidence  associated  with  the  quality  of  the 
technical  evaluations  of  offerors’  proposals.  There  were  no  previously-validated 
scales  available  for  this  factor.  The  variable  consisted  of  six  questions  using  a 
seven-point  Likert-type  scale  with  anchors  of  strongly  agree  and  strongly  disagree. 
Three  variables  from  this  construct  were  discarded  as  a  result  of  factor  analysis. 

Table  8.  Construct  Measurement  of  Technical  Evaluation  Effectiveness 

Scale  Item _ Survey  Question _ 

TEE1#* *  At  least  once,  a  technical  evaluator  was  required  to  change  the  wording  of  his  or  her 

technical  evaluations. 

TEE2*  At  least  one  technical  evaluator  expressed  concern  about  not  being  able  to  say  what 

needs  to  be  said  in  the  technical  evaluation. 

TEE3*  At  least  one  technical  evaluator  was  concerned  that  the  constraints  imposed  on  his 

or  her  evaluations  impeded  the  evaluator’s  ability  to  write  a  meaningful  evaluation. 

TEE4#*  The  technical  evaluators  believed  that  the  quality  of  their  evaluations  could  not  have 

been  better. 

TEE5#*  If  there  were  no  Federal  Acquisition  Regulations,  no  source  selection  policy,  and  no 

threat  of  a  bid  protest,  the  quality  of  the  technical  evaluations  would  have  been  the 
same. 

TEE6*  Upon  evaluation  of  proposals,  at  least  one  technical  evaluator  expressed  a  need  to 

change  at  least  one  evaluation  criterion  or  its  definition. 

*  item  was  discarded  due  to  Exploratory  Factor  Analysis 

*  item  was  reverse  coded 
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Requirement  criticality/importance  is  the  level  of  significance  of  a  particular 
acquisition  at  the  macro  level.  The  requirement  criticality/importance  factor  was 
validated  by  qualitative  interviews.  Table  9  shows  the  five  questions  that  were  used 
to  measure  the  acquisition’s  importance  with  regard  to  the  overall  mission  of  the 
agency.  The  questions  were  formulated  from  studies  conducted  on  procurements  in 
business-to-business  markets  (Schoenherr  &  Mabert,  2011).  The  variable  used  a 
seven-point  Likert-type  scale  with  anchors  of  strongly  agree  and  strongly  disagree. 
Two  variables  from  this  construct  were  discarded  as  a  result  of  factor  analysis. 


Table  9. 

Construct  Measurement  of  Requirement  Criticality/Importance 

Scale  Item 

Survey  Question 

RCI1# 

This  requirement  was  important  for  the  good  operation  of  our  customer’s 
organization. 

RCI2 

This  requirement  supported  a  core  competency  of  our  customer’s 
organization. 

RCI3 

Compared  to  other  purchases  for  this  customer,  this  requirement  was 
important. 

RCI4* 

An  unsuccessful  outcome  of  the  RFP  would  have  had  only  minor 
consequences  to  our  customer 

RCI5# 

As  a  portion  of  the  customer’s  total  annual  spending  amount,  the  dollar  value 
of  this  requirement  was  high. 

RCI6 

As  a  portion  of  the  customer’s  total  annual  spending  amount,  the  dollar  value 
of  this  requirement  was  high. 

#  item  was  discarded  due  to  Exploratory  Factor  Analysis 

*  item  was  reverse  coded 


Buyer  satisfaction  represents  the  level  of  contentment  the  agency  has  with 
the  contractor’s  performance  that  won  the  source  selection  in  question.  The  buyer 
satisfaction  factor  was  validated  by  qualitative  interviews.  Table  10  shows  the  five 
questions  that  were  constructed  to  measure  the  level  of  satisfaction  that  resulted 
from  the  source  selection  being  surveyed.  The  scale  was  adapted  from  a  study 
conducted  on  buyer-seller  relationships  in  commercial  markets  (Cannon  &  Perreault, 
1999).  The  variable  consisted  of  five  questions  using  a  seven-point  Likert-type  scale 
with  anchors  of  strongly  agree  and  strongly  disagree.  One  variable  from  this 
construct  was  discarded  as  a  result  of  factor  analysis. 
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_ Table  10.  Construct  of  Buyer  Satisfaction _ 

Scale  Item  Survey  Question _ 

BS1#*  Our  customer  regrets  the  decision  to  do  business  with  this  contractor. 

BS2  Overall,  our  customer  is  very  satisfied  with  this  contractor. 

BS3  Our  customer  is  very  pleased  with  what  the  contractor  does  for  the  customer. 

BS4*  Our  customer  is  not  completely  happy  with  this  contractor. 

BS5#  If  we  had  to  do  it  all  over  again,  we  would  still  choose  to  use  this  contractor. 

#  item  was  discarded  due  to  Exploratory  Factor  Analysis 

*  item  was  reverse  coded 

Contractor  performance  is  the  overall  measure  of  the  successful  offeror’s 
adherence  to  contract  obligations.  Vetting  from  industry  practitioners  validated  the 
contractor  performance  construct.  Table  1 1  shows  the  seven  aspects  of  contract 
execution  that  were  used  to  measure  the  level  of  contractor  performance  that 
resulted  from  the  contracts  awarded  from  the  source  selections  for  which 
respondents  answered  the  survey  questions.  The  scale  was  adapted  from  past 
marketing  and  operations  management  studies  (Fawcett,  Smith,  &  Cooper,  1997; 
Cannon,  Achrol,  &  Gundlach,  2000;  Prahinski  &  Benton,  2004).  The  variable 
consisted  of  five  questions  using  a  seven-point  Likert-type  scale  with  anchors  of 
needs  improvement  and  superior  performance. 

_ Table  1 1 .  Construct  Measurement  of  Contractor  Performance 

Scale  Item _ Survey  Question _ 

CP1  Product/service  quality  per  specifications 

CP2  Delivery  performance  per  specifications 

CP3  Product/service  consistently  meets  customer  expectations 

CP4  Responsiveness  to  requests  for  changes 

CP5  Required  service  and/or  technical  support 

CP6  Non-conformance  rate 

CP7  Overall  performance 


Vetting  from  industry  practitioners  validated  the  protest  risk  construct.  Protest 
risk  was  an  exploratory  construct  that  consists  of  two  parts  since  risk  is  comprised  of 
the  product  of  the  magnitude  of  the  consequences  and  the  probability  of  occurrence. 
T able  1 2  shows  the  questions  used  to  assess  the  desirability  of  each  of  five 
consequences  and  each  of  their  associated  probabilities  of  occurring.  The  five 
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consequences  were  validated  by  interview  informants  to  be  those  most  likely  to 
occur  and  those  most  abhorred.  Risk  was  calculated  by  multiplying  the 
consequences’  probability  of  occurrence  by  the  desirability  of  the  consequence.  The 
result  yielded  an  overall  protest  risk  score.  The  construct  consisted  of  five  possible 
consequences  using  a  Likert-type  scale  with  anchors  of  completely  undesirable  and 
completely  desirable,  and  then  listed  the  same  five  consequences  using  a  probability 
of  occurance  scale  with  anchors  of  0  percent  to  100  percent. 

_ Table  12.  Construct  Measurement  of  Protest  Risk _ 

Scale  Item  Survey  Question _ 

PR1  Increased  costs  to  settle  a  terminated  contract(s). 

PR2  Time  delay  to  the  mission. 

PR3  Embarrassment/shame. 

PR4  Increase  in  workload  to  resolve  the  protest. 

PR5  Career  repercussions  for  making  a  mistake  or  omission  that  caused  a  bid 

protest. 


risk  =  probability  of  occurrence  x  magnitude  of  consequences 


Source  selection  method  appropriateness  is  the  perceived  extent  that  the 
chosen  source  selection  method  fits  the  requirement,  the  goals  of  the  source 
selection,  the  commercial  market,  and  the  acquisition  situation.  The  source  selection 
method  appropriateness  factor  is  new.  The  factor  was  validated  by  qualitative 
interviews.  The  decision  about  the  source  selection  method  to  be  used  is  made  very 
early  on  in  the  acquisition.  The  variable  consisted  of  six  questions  using  a  seven- 
point  Likert-type  scale  with  anchors  of  strongly  disagree  and  strongly  agree,  shown 
in  Table  13. 
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_ Table  13.  Source  Selection  Method  Appropriateness _ 

Scale  Item _ Survey  Question _ 

SSMA1  Our  acquisition  strategy  was  the  best  means  to  source  our  requirement. 

SSMA2  Our  acquisition  strategy  was  the  best  means  to  achieve  our  acquisition 

objectives. 

SSMA3  It  would  have  been  difficult  to  achieve  our  goals  without  the  use  of  our 

acquisition  strategy. 

SSMA4  The  source  selection  method  we  used  (e.g..,  LPTA,  full-trade-off,  or  PPT) 

was  the  most  appropriate  for  this  requirement. 

SSMA5  Our  acquisition  strategy  ensured  we  selected  the  best  offeror. 

SSMA6  Our  acquisition  strategy  provided  the  best  fit  to  the  buying  situation  (e.g., 

complexity,  dollar  value,  acquisition  objectives,  contract  length,  performance 
risk,  criticality  to  the  mission,  availability  of  supply,  time  available  to  award  a 

The  source  selection  method  fit  variable  is  an  exploratory  concept  that  had 
not  been  previously  tested.  Answers  to  the  questions  in  Table  14  were  used  to 
explore  various  aspects  of  the  overall  source  selection.  This  is  a  more  objective 
measure  of  the  same  construct  in  Table  13,  source  selection  method 
appropriateness.  Source  selection  method  fit  measures  the  difference  in  time  and 
resources  used,  method  used,  and  satisfaction  with  discussions.  Measures  for  each 
question  are  listed  at  the  bottom  of  Table  14. 
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Table  14.  Source  Selection  Method  Fit 


Scale  Item 

Survey  Question 

SSMF1* 

During  acquisition  planning,  how  many  days  were  planned  in  the  milestones 
from  receipt  of  a  complete  requirements  package  to  award  of  the  contract(s)? 

SSMF2* 

Number  of  actual  days  from  receipt  of  a  complete  requirements  package  to 
award  of  the  contract: 

SSMF3* 

Number  of  people  on  the  source  selection  team  including  all  advisors, 
reviewers,  full-time  participants,  and  part-time  participants: 

SSMF4* 

Absent  a  risk  of  a  bid  protest,  in  your  opinion,  how  many  people  ideally  should 
have  been  on  the  source  selection  team: 

SSMF5** 

What  source  selection  method  was  used?  (Select  one.) 

SSMF6*** 

Rate  the  appropriateness  of  each  source  selection  method  for  the 
requirement  on  a  scale  of  1  to  7,  where  1  represents  “completely 
inappropriate”  and  7  represents  “completely  appropriate.”  LPTA  (Lowest  Price 
Technically  Acceptable) 

SSMF7*** 

PPT  (Price-Past  Performance  Trade-off ) 

SSMF6*** 

Full  Trade-off 

SSMFa**** 

Rate  your  level  of  satisfaction  with  the  freedom  to  openly  discuss  those 
aspects  of  the  proposals  that  needed  to  be  discussed  with  the  offeror  in  order 
to  fully  understand  the  offer  and  to  properly  evaluate  the  proposals. 

SSMF10***** 

Hypothetically,  if  there  was  no  ability  to  protest,  rate  the  extent  that 
discussions  with  offerors  would  have  differed. 

SSMF11****** 

Did  the  RFP  advise  offerors  that  the  government  intended  to  award  without 
discussions? 

SSMF12****** 

Did  you  conduct  discussions? 

SSMF13*** 

Considering  the  risk,  criticality,  dollar  value,  contribution  to  the  mission,  and 
complexity,  rate  the  appropriateness  of  awarding  a  contract  without 
conducting  discussions. 

SSMF14****** 

Did  offerors  make  oral  presentations? 

SSMF15*** 

Considering  the  risk,  criticality,  dollar  value,  contribution  to  the  mission,  and 
complexity,  rate  the  appropriateness  of  not  utilizing  oral  presentations. 

*  Measured  by  text  input  into  a  field 
**  Used  a  drop  box  with  appropriate  choices 

***  Likert-type  scale  with  anchors  of  completely  appropriate  and  completely  inappropriate 
****  Likert-type  scale  with  anchors  of  completely  satisfied  and  completely  dissatisfied 
*****  Likert-type  scale  with  anchors  of  no  difference  and  substantial  difference 
******  Choices  of  Yes  or  No 

The  number  of  awarded  contracts  variable  was  validated  by  qualitative 
interviews.  The  measure  was  used  to  analyze  the  difference  between  the  number  of 
contract  awards  that  were  planned  and  the  number  of  contracts  actually  awarded. 
These  were  objective  measures  along  a  continuous  scale.  The  questions  used  are 
shown  in  Table  15. 
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Table  15.  Measurement  of  Number  of  Awarded  Contracts 


Scale  Item _ Survey  Question _ 

AC1  According  to  your  acquisition  planning,  how  many  contracts  were  originally 

intended  to  be  awarded? 

AC2_ How  many  contracts  were  actually  awarded? 


The  transaction  costs  variable  is  an  objective  measure  that  was  used.  This 
measure  consisted  of  25  items,  shown  in  Table  16  and  was  validated  through 
qualitative  interviews  and  discussions  with  other  industry  practitioners.  The  first  10 
items  gathered  the  number  of  iterations  of  common  source  selection  documents. 
They  were  represented  by  the  abbreviation  TCI.  The  last  15  attempted  to  quantify  in 
dollars  the  amount  of  time  spent  on  the  source  selection  by  each  member  of  the 
team,  and  were  represented  by  the  abbreviation  TCP.  The  roles  were  assigned 
using  information  from  the  DOD  source  selection  procedures  (OUSD  [AT&L],  201 1). 
There  was  no  previously  validated  scale  to  measure  these  transaction  costs.  Text 
boxes  to  gather  data  were  used  to  answer  these  questions. 
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Table  1 6.  Construct  of  T ransaction  Costs 

Survey  Question 


Scale  Item 

TCI1  During  the  source  selection,  how  many  iterations  of  each  of  the  following  documents 

were  generated? 

Source  selection  decision  document 
TCI2  Comparative  assessment/proposal  analysis  report 

TCI3  Evaluation  notices  (sum  of  iterations  for  all  ENs  across  all  offerors) 

TCI4  Source  selection  plan 

TCI5  Debriefing  scripts  (sum  of  iterations  for  all  scripts  across  all  offerors) 

TCI6  Technical  evaluations  (sum  of  iterations  for  all  evaluations  across  all  offerors) 

TCI7  Past  performance  evaluations  (sum  of  iterations  for  all  evaluations  across  all 

offerors) 

TCI8  Cost/price  analysis 

TCI9  Color/rating  chart 

TCI10  Evaluation  briefing  charts  for  reviewers  and  SS 

TCP1  For  the  following  roles,  please  provide  the  number  of  people  within  that  role,  their 

grade(s)/rank(s),  and  the  fraction  of  1  year’s  time.  If  no  involvement,  just  skip  to  the 
next  role.  Contracting  Officer 

TCP2  Contract  specialists 

TCP3  Technical  Evaluators 

TCP4  Cost/price  analysts 

TCP5 

Supervisors 


TCP6 

TCP7 

TCP8 

TCP9 

TCP10 

TCP11 

TCP12 

TCP13 

TCP14 

TCP15 


Legal 

Past  performance  team 

Source  selection  advisory  council/non-legal  advisors 

Source  selection  authority 

Financial  management 

Program  manager/requirements  office 

Small  business  representative 

Contracted  consultants/labor 

Other  (description) 

Other  (data  field) 


The  competence  factor  was  validated  by  qualitative  interviews  and 
discussions  with  industry  practitioners  as  shown  in  Table  17. 
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_ Table  17.  Construct  Measurement  of  Contracting  Officer  Competence 

Scale  Item _ Survey  Question _ 

C1  How  many  total  years  of  experience  do  you  have  in  contracting? 

C2  In  how  many  source  selections  have  you  participated  throughout  your  career? 


E.  SURVEY  PRE-TEST 

Six  industry  practitioners,  which  consisted  of  graduate-level  students  and 
professors  who  specialize  in  DOD  contracting  tested  the  initial  survey.  Feedback 
received  was  used  to  refine  questions  and  limit  survey  length.  Responses  supported 
the  proposed  research  hypothesis,  but  in  order  to  shorten  the  survey,  one 
antecedent — risk  tolerance —  was  removed. 

Once  constructed,  the  order  of  the  survey  questions  was  structured  to  reduce 
bias  among  scale  items  by  mixing  questions  with  like  scales  and  scale  anchors.  The 
complete  survey  can  be  found  in  Appendix  A.  As  required,  the  survey  was  reviewed 
and  approved  by  the  Navy  Survey  Office.  An  additional  review  was  conducted 
through  the  Department  of  the  Navy’s  Institutional  Review  Board  to  ensure  the 
protection  of  human  subjects. 

F.  RESEARCH  DESIGN 

This  study  attempts  to  answer  the  following  questions  via  qualitative  and 
quantitative  methods: 

•  Does  a  fear  of  protest  exist  in  the  acquisition  workforce? 

•  If  so,  what  are  the  contributing  factors? 

•  If  so,  what  are  the  outcomes? 

A  conceptual  model  that  included  16  variables  was  developed  by  the 
research  team  to  identify  those  factors  that  contributed  to  protest  fear  and 
hypothesize  what  verifiable  results  could  be  observed.  The  model  was  then  vetted 
by  discussion  with  22  current  and  previously-warranted  contracting  officers  and 
three  Air  Force  and  Navy  contracting  unit  leaders  for  content,  or  face,  validity. 

To  precisely  identify  and  measure  the  relationships  shown  in  the  conceptual 
model,  quantitative  analysis  through  hypothesis  testing  between  each  contributing 
and  resulting  variable  and  fear  of  protest  was  performed.  The  path  diagram  of  the 
conceptual  model  (see  Figure  1)  contains  14  relationships  that  represent  the  14 
hypotheses.  The  relationships  between  variables  are  depicted  using  arrows;  “the 
directionality  of  causal  paths  begins  from  a  variable  at  the  arrow’s  tail  and  ends  at  a 
variable  about  the  arrow’s  point”  (Muir,  2010,  p.  38).  These  arrows  contain  a  plus  or 
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minus  sign  to  reflect  the  expected  direction  of  the  relationship.  The  primary  methods 
used  to  perform  statistical  analysis  are  simple  and  multiple  linear  regressions. 

Linear  regression  tries  to  model  the  relationship  between  two  variables  by 
fitting  a  linear  equation  to  data  on  the  X-  and  Y-axis.  “The  case  of  one  dependent 
and  independent  variable  is  called  simple  linear  regression.  For  more  than  one 
explanatory  variable,  it  is  called  multiple  linear  regression”  (“Linear  Regression,” 
1998). 

In  linear  regression,  data  is  modeled  “using  linear  predictor  functions,  and 
unknown  model  parameters  are  estimated  from  the  data”  (Stanton,  2001,  p.  237). 
The  result  is  known  as  a  linear  model.  Most  commonly,  “linear  regression  refers  to  a 
model  in  which  the  conditional  mean  of  Y  given  the  value  of  X  is  an  affine  function  of 
X”  (Cohen,  West,  &  Aiken,  2003,  p.  42).  The  purpose  of  causal  modeling  is  to  predict 
or  explain  phenomenon.  Causal  models  are  “appropriate  for  domains  in  which  many 
modeled  events  can  be  conceptualized  as  processes  causing  other  events  which  in 
turn  trigger  other  processes”  (Lemmer,  1993,  p.  143). 

Multiple  regression  is  a  form  of  causal  modeling  that  can  be  used  to  analyze 
the  relationship  between  one  dependent  variable  and  several  independent  variables. 
The  “flexibility  and  adaptability  of  multiple  regression  allow  its  use  with  almost  any 
dependence  relationship”  (Hair,  Black,  Babin,  &  Anderson  p.  166).  Multiple 
regression  can  be  made  applicable  to  a  wide  range  of  problems  because  of  this 
flexibility.  Its  application  is  appropriate  for  two  types  of  research  problems:  prediction 
and  explanation.  Prediction  deals  with  the  magnitude  that  the  independent  variable 
can  predict  the  value  of  the  dependent  variable.  Explanation  uses  regression 
coefficients  for  the  independent  variables  and  “attempts  to  develop  a  substantive  or 
theoretical  reason  for  the  effects  of  the  independent  variables”  (Hair  et  al. ,  p.  167). 

The  goal  of  this  research  is  not  to  predict  the  level  of  the  fear  dependent 
variable,  but  to  explain  the  significance  of  its  relationships  with  the  independent 
variables  that  are  used.  Multiple  regression  is  the  appropriate  tool  to  answer  the 
questions  being  asked  because  it  can  objectively  asses  the  magnitude  and 
character  of  the  relationships  of  the  conceptual  model.  When  regression  analysis  is 
performed,  it  is  assumed  that  there  will  be  a  linear  association  between  the 
dependent  and  independent  variables.  With  multiple  regression,  “transformations  or 
additional  variables  are  available  to  assess  whether  other  types  of  relationships 
exist,  particularly  curvilinear  relationships.  This  flexibility  ensures  that  the  researcher 
may  examine  the  true  nature  of  the  relationship  beyond  the  assumed  linear 
relationship”  (Hair  et  al.,  p.  1 68). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  42  - 

NAVAL  POSTGRADUATE  SCHOOL 


G. 


CONCLUSION 


In  this  chapter,  the  qualitative  interviews  that  occurred  were  discussed.  Next, 
the  population  and  sampling  methods,  followed  by  the  survey  design  and  construct 
measurement  was  discussed.  Finally,  the  research  design  and  statistical  method 
used  for  the  data  analysis  was  discussed.  In  the  next  chapter,  the  results  of  the  data 
collection  and  statistical  analysis  are  presented. 
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IV.  RESULTS 


A.  INTRODUCTION 

This  chapter  presents  the  results  of  the  survey,  hypothesis  testing,  and 
conceptual  model.  The  data  collection  results  are  discussed  first,  followed  by  the 
sample  demographics.  The  results  of  tests  for  normality,  outliers,  non-response  bias, 
reliability,  and  validity  are  then  discussed.  Next,  the  results  of  the  exploratory  factor 
analysis  (EFA)  are  performed  in  SPSS  Statistics  Version  22.0.  The  results  of  the 
regression  analyses  performed  are  then  discussed.  Finally,  the  conceptual  model 
with  all  hypothesized  relations  is  presented. 

When  analyzing  the  significance  of  the  relationship  between  constructs,  the 
path  coefficient  is  used  to  measure  strength.  Alpha  levels  that  are  less  than  .05  are 
considered  to  have  a  strong  relationship  between  dependent  and  independent 
variables. 

B.  DATA  COLLECTION 

Data  was  collected  from  a  NPS-approved  web-based  survey  tool,  Lime 
survey.  A  survey  invitation  (Appendix  B)  was  sent  via  e-mail  to  contracting  officers 
who  had  executed  a  FAR  Part  15-based  formal  source  selections  with  a  dollar  value 
greater  than  or  equal  to  $150,000  during  FY2013.  E-mail  addresses  were  extracted 
from  the  FPDS-NG  database.  A  memorandum  of  support  from  the  Deputy,  Assistant 
Secretary  of  the  Navy  (Acquisitions  and  Procurement)  was  referred  to  in  the  e-mail 
to  notify  the  respondents  of  sponsorship  (Appendix  C).  The  survey  was  left  active  for 
42  days,  with  three  reminder  invitations  being  sent  during  that  time.  The  survey  was 
sent  to  a  population  of  4,279  unique  e-mail  addresses.  Of  these  unique  addresses, 
260  undeliverable  responses  were  received,  and  137  individuals  replied  stating  that 
they  had  not  participated  in  a  FAR  Part  15  Source  Selection.  As  a  result,  the  total 
population  was  reduced  to  3,882.  At  the  survey  closure,  there  were  661  responses 
received,  which  yielded  a  17.02  percent  response  rate.  Of  the  responses  received, 

31 1  had  to  be  deleted  because  of  missing  or  invalid  data,  leaving  350  usable 
responses.  The  final  response  rate  of  usable  responses  is  9.01  percent. 

While  the  response  rate  is  low,  it  is  not  unlike  that  of  other  published  business 
research.  In  a  study  focused  on  declining  response  rates  in  supply  chain 
management  research,  Melnyk,  Page,  Wu,  and  Burns  (2012)  stated  that  “the  issue 
of  declining  response  rates  is  a  growing  concern”  (p.  35).  Their  research  showed  a 
sharp  decline  starting  in  2002,  with  a  steady  decline  of  one  percent  annually.  Five 
top  journals  reported  low  end  survey  response  rates  ranging  from  three  percent  to 
eight  percent.  Survey  length  is  thought  to  be  a  one  of  the  key  contributors  to  the 
decline.  Melnyk  et  al.  further  stated  that  “for  every  additional  question  over  20,  the 
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researcher  can  expect  the  response  rate  to  fall  by  0.12  percentage  points. 
Consequently,  the  researcher  can  expect  the  response  to  fall  by  over  seven 
percentage  points  in  response  to  an  80  question  survey.”  (Melnyk  et  al.,  2012,  p.  43) 
The  summary  descriptive  statistics  from  the  remaining  350  usable  responses  can  be 
found  in  Table  18. 
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Table  18.  Summary  Statistics 


Variable 

Measure 

N 

Mininiitm 

Maximum 

Mean 

Ski.  Deviation 

Variable 

Measure 

N 

Minimum 

Maximum 

Mean 

Ski.  Deviation 

Fear 

FEAR1 

350 

1 

7 

437 

1.919 

Protest  Risk 

PR1 

350 

1 

11 

3.44 

2.86 

FEAR  2 

350 

1 

7 

4.22 

1.887 

PR2 

350 

1 

11 

2.53 

2294 

FEAR3 

350 

1 

7 

4.41 

1.959 

PR3 

350 

1 

11 

5.42 

3.114 

FEAR4 

350 

1 

7 

3.82 

1.765 

PR4 

350 

1 

10 

3.51 

2.87 

FEAR5 

350 

1 

7 

333 

1.666 

FR5 

350 

1 

11 

4.63 

3215 

FEAR6 

350 

1 

7 

338 

1.708 

PR1 

350 

0.00% 

100.00% 

48.17% 

31.01% 

PR2 

350 

0% 

100% 

75.09% 

2825% 

Contracting  Officer  Authority 

CO  AUTH1 

350 

1 

7 

5.27 

1.546 

PR3 

350 

0.00% 

100.00% 

36.09% 

31.57% 

CO  AUTH2 

350 

1 

7 

5.43 

1.452 

PR4 

350 

0% 

100% 

74.26% 

28.55% 

CO  ATJTH3 

350 

1 

7 

5.6 

1.341 

PR5 

350 

0.00% 

100.00% 

33.23% 

3027% 

COAUTH4 

350 

1 

7 

3.84 

1.703 

Source  Selection 

CO  AUTH5 

350 

1 

7 

4.17 

1.887 

Method  Appropn  ateness 

SSMA1 

350 

1 

7 

5.51 

1315 

CO  ATJTT16 

350 

1 

7 

3.1 

1.759 

SSMA2 

350 

1 

7 

5.61 

1243 

SSMA3 

350 

1 

7 

5.17 

1.532 

PALT  Planned 

PALT  PI 

350 

1 

7 

438 

1.795 

SSMA4 

350 

1 

7 

5.76 

1292 

PALT  P2 

350 

1 

7 

336 

1.761 

SSMA5 

350 

1 

7 

5.52 

1339 

PALT  P3 

350 

1 

7 

3.83 

1.75 

SSMA6 

350 

1 

7 

5j51 

1395 

SSMA7 

350 

0 

1000000 

3168.97 

53702.779 

Technical  Evaluation  Effectiveness 

TEE1 

350 

1 

7 

533 

1.584 

SSMA8 

350 

0 

200 

2.89 

14252 

TEE2 

350 

1 

7 

3.5 

1.714 

SSMA9 

350 

0 

150 

3.04 

13.574 

TEE3 

350 

1 

7 

337 

1.628 

Source  Selection 

TEE4 

350 

1 

7 

437 

1.434 

Method  Fit 

SSMF1 

350 

0 

780 

182.97 

131.168 

TEE5 

350 

1 

7 

3.03 

1.817 

SSMF2 

350 

0 

3290 

236.72 

244.081 

'IEE6 

350 

1 

7 

3.47 

1.744 

SSMF3 

350 

0 

138 

8.96 

10.052 

SSMF4 

350 

0 

130 

7.65 

9.053 

Reqmemeni  Criticality /I  nrportance 

RCI1 

350 

1 

7 

5.9 

1.093 

SSMF5 

350 

1 

3 

2.12 

0.931 

RCI2 

350 

1 

7 

5.79 

1.135 

SSMF6 

350 

1 

7 

4.16 

2.181 

RCI3 

350 

1 

7 

5.77 

1.142 

SSMF7 

350 

1 

7 

3.82 

1.776 

RCI4 

350 

1 

7 

2.41 

1.423 

SSMF8 

350 

1 

7 

4.89 

2.061 

RCI5 

350 

1 

7 

4.15 

1.666 

SSMF9 

350 

1 

7 

328 

1.718 

RCI6 

350 

1 

7 

4.15 

1.668 

SSMF10 

350 

1 

7 

337 

2.041 

SSMF13 

350 

1 

7 

4.2 

2.113 

Buyer  Satisfaction 

BS1 

350 

1 

7 

2.44 

1.359 

SSMF15 

350 

1 

7 

4.7 

2.099 

BS2 

350 

1 

7 

5.5 

1.232 

Actual  Number  of 

BS3 

350 

1 

7 

5.46 

1.176 

Contracts  Awarded 

AC1 

350 

0 

60 

224 

4.879 

BS4 

350 

1 

7 

3.02 

1.474 

AC2 

350 

0 

55 

239 

5.506 

BS5 

350 

1 

7 

5.61 

1.225 

Contractor  Performance 

CPI 

350 

1 

7 

4.69 

1.372 

CP2 

350 

1 

7 

4.74 

132 

CP3 

350 

1 

7 

4.74 

132 

CP4 

350 

1 

7 

4.76 

1.303 

CP5 

350 

1 

7 

4.75 

1.261 

CP6 

350 

1 

7 

433 

1.213 

CP7 

350 

1 

7 

4.88 

1.223 
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c. 


SAMPLE  DEMOGRAPHICS 


From  the  350  usable  responses,  the  average  respondent  had  13.6  years  of 
federal  contracting  experience.  There  were  314  respondents  who  elected  to  identify 
their  gender.  Results  were  even  with  male  respondents  accounting  for  50.64  percent 
and  female  respondents  accounted  for  49.36  percent.  The  majority  of  respondents 
represented  had  at  least  a  bachelor’s  degree.  Only  2.29  percent  of  respondents  held 
a  high  school  diploma  or  general  equivalency  diploma,  3.14  percent  held  an 
associate’s  degree,  38  percent  held  a  bachelor’s  degree,  54  percent  held  a  master’s 
degree,  and  only  2.57  percent  of  respondents  held  a  doctorate  degree.  As  to  be 
expected,  respondents’  Acquisition  Professional  Development  Program  (APDP) 
certification  levels  in  contracting  were  skewed  to  higher  levels.  Typically,  more 
experienced  contracting  professionals  conduct  FAR  Part  15  source  selections.  Only 
1 .43  percent  of  the  respondents  held  no  APDP  contracting  certification,  2.86  percent 
held  a  Level  I  APDP  certification,  37.43  percent  held  a  Level  II  APDP  certification, 
and  58.29  percent  held  a  Level  III  APDP  certification.  Demographics  for  the 
education  and  certification  levels  of  the  respondents  can  be  found  in  Table  19. 


Table  19.  Education  and  Certification  Demographics 


Education  Level 

%  of  T  otal 

Certification  Level 

%  of  Total 

High  School  Diploma  of  GED 

2.29% 

No  Certification 

1 .43% 

Associate’s  Degree 

3.14% 

Level  1 

2.86% 

Bachelor’s  Degree 

38.00% 

Level  II 

37.43% 

Master’s  Degree 

54.00% 

Level  III 

58.29% 

Doctoral  or  professional 
degree 

2.57% 

Respondents  were  asked  to  refer  back  to  the  most-recent  completed  source 
selection  above  $150,000  in  which  they  had  participated.  The  range  of  contract  type 
and  type  of  products/services  are  reported  (see  Table  21).  Of  the  respondents,  60 
percent  reported  the  use  of  fixed-priced,  27  percent  reported  cost  reimbursement, 
one  percent  reported  Time  and  Materials,  one  percent  reported  labor-hour,  nine 
percent  reported  hybrid,  and  two  percent  reported  as  other.  The  type  of  supply  or 
service  bought  was  also  reported.  Services  were  bought  in  58  percent  of  the  cases, 
construction  made  up  17  percent  of  the  purchases,  supplies  or  commodities  made 
up  16  percent,  weapons  systems  made  up  seven  percent,  and  other  capital 
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equipment  made  up  three  percent.  Experience-related  demographics  information 
can  be  found  in  Table  20. 


Table  20. 

Experience 

Years  of  Experience 

Total 

Other  Statistics 

Total 

Average  of  all  respondents 

13.12 

Avg  Number  of  Source 
Selections 

18.31 

Respondents  with  5  or  less 
years 

110  (31.4%) 

Avg  Number  of  Protests 
experienced 

1.96 

Respondents  with  between  5 
and  15  years 

130  (37.1%) 

Number  (%)  who  had  a 
protest  sustained 

54  (15.4%) 

Respondents  with  greater 
than  15  years 

120  (34.3%) 

Table  21. 

Contract  Type  and  Item/Service  Procured 

Contract  Type  %  of  T otal 

Supply  or  Service 
Purchased 

%  of  Total 

Fixed  Price 

60.00% 

Services 

58.00% 

Cost  Reimbursement 

27.14% 

Construction 

16.86% 

Time  and  Materials 

1.14% 

Supplies  or  Commodities 

15.71% 

Labor  Hours 

0.57% 

Weapons  System 

6.57% 

Hybrid 

9.14% 

Other  Capital  Equipment 

2.86% 

Other 

2.00% 

D.  ASSUMPTIONS 

1.  MISSING  DATA 

Out  of  661 ,  there  were  31 1  responses  discarded  because  of  missing  data, 
leaving  a  total  of  350  usable  responses.  Of  these  responses,  36  did  not  include  a 
properly  reported  gender  answer;  these  respondents  were  not  included  in  the 
gender  demographics  reporting.  The  Transaction  Cost  variable  had  multiple  missing 
and  inconsistent  items  of  data  since  this  was  an  onerous  task  for  respondents  to 
complete,  and  since  it  was  an  optional  part  of  the  survey.  Additionally,  the  survey 
collected  the  pay  grade  and  allocation  of  time  via  open  text  fields  enabling 
inconsistent  response  formats.  All  reporting  of  this  data  includes  only  the  complete 
responses  with  unambiguous  data.  Only  270  responses  were  able  to  be  used  for  this 
analysis. 
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2. 


OUTLIERS 


Outliers  are  classified  as  “a  unique  combination  of  characteristics  identifiable 
as  distinctly  different  from  other  observations”  (Hair  et  al.,  p.  63).  Outliers  such  as 
procedural  error,  extraordinary  events,  and  extraordinary  observations  can  arise  for 
a  variety  of  reasons.  .  Most  of  the  questions  contained  in  this  survey  were  on  a 
Likert-type  scale  where  the  ranges  of  responses  were  limited. 

The  presence  of  Univariate  and  Multivariate  outliers  was  tested  using  SPSS 
version  22.  To  conduct  the  Univariate  test,  the  score  of  10  variables  were  converted 
to  standardized  values,  or  z-scores.  Then,  the  standard  value  was  evaluated  against 
a  benchmark  in  which  potential  outliers  had  the  z-score  of  ±3.00  or  higher  (Muir, 
2010,  p.  56).  The  range  of  standard  scores  can  be  seen  in  Table  22.  Five  variables 
had  cases  that  were  above  or  below  the  threshold.  Contracting  officer  authority  had 
five  cases  that  were  outside  of  the  threshold;  Requirement  Criticality  or  Importance 
had  three  cases  that  were  outside  of  the  threshold;  Buyer  Satisfaction  had  four 
cases  that  were  outside  of  the  threshold;  Contractor  Performance  had  three  cases 
that  were  outside  of  the  threshold;  and  Source  Selection  Method  Appropriateness 
had  seven  cases  that  were  outside  of  the  threshold.  Each  case  was  examined  for 
inconsistent  responses.  This  examination  did  not  lead  to  any  responses  being 
discarded. 

Since  multivariate  detection  involves  several  variables,  the  ability  to  measure 
the  multidimensional  position  of  each  response  relative  to  a  common  point  is 
needed.  The  Mahalanobis  Distance,  or  D2,  is  a  multivariate  assessment  of  each 
observation  across  a  set  of  variables  (Hair  et  al.,  2010).  The  same  10  variables  were 
used  to  calculate  the  D2in  SPSS.  The  critical  value  was  29.59  when  10  variables 
were  used.  Five  cases  exhibited  a  D2  score  higher  than  29.59.  Each  case  was 
examined  for  inconsistencies  or  incoherent  responses.  This  examination  also  led  to 
no  responses  being  discarded.  The  Transaction  Cost  document  iteration  variable 
was  also  examined  for  incoherent  responses.  Twelve  outlying  responses  were 
truncated  to  the  highest  reasonable  score.  “One  alternative  to  transformation  is 
truncation,  wherein  extreme  scores  are  recorded  to  the  lowest  reasonable  score” 
(Osborne  &  Overbay,  2004,  p.  2).  Reasonable  scores  were  determined  using  the 
average  number  of  iterations  for  each  document.  If,  for  example,  a  response 
indicated  that  the  respondents  experienced  99  iterations  of  a  document,  this  was 
considered  an  unrealistic  response.  This  response  was  then  truncated  to  the 
average  iteration  number  for  the  subject  document. 
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Table  22.  Univariate  Range  of  Standard  Scores 


Variable 

N 

Minimum 

Maximum 

Zscore(FEAR) 

350 

-1.88982 

1.96546 

Zscore(CO_AUTH) 

350 

-3.45073 

1.13423 

Zscore(PALT_P) 

350 

-1.73510 

2.04099 

Zscore(TEE) 

350 

-1.75669 

2.55294 

Zscore(RCI) 

350 

-4.63729 

1.18532 

Zscore(BS) 

350 

-4.72418 

4.14633 

Zscore(CP) 

350 

-3.19421 

1.94928 

Zscore(SSMA) 

350 

-4.06658 

1.34293 

3.  Normality 

Normality  is  the  most  fundamental  assumption  in  multivariate  analysis  and  the 
benchmark  for  statistical  methods.  If  the  variation  from  the  normal  distribution  is 
sufficiently  large,  all  resulting  statistical  tests  are  invalid  (Hair  et  al.,  2010).  The 
shape  of  normally  distributed  data  is  a  symmetric  bell-shaped  curve.  A  distribution 
can  be  described  by  two  basic  measures,  skewness  and  kurtosis.  Skewness 
describes  the  lateral  balance  of  the  distribution.  A  positive  skew  describes  a 
distribution  that  is  shifted  left,  while  a  negative  skew  is  shifted  to  the  right.  Kurtosis 
denotes  the  peak  or  flatness  of  a  distribution  when  compared  to  a  normal  bell 
shaped  distribution.  Distributions  with  a  higher,  or  more  peaked,  distribution  are  said 
to  leptokurtic,  while  flatter  distributions  are  said  to  be  platykurtic  (Hair  et  al.,  2010). 

Graphical  analysis  of  a  distribution  using  a  histogram  is  the  easiest  test  for 
normality.  However,  this  method  can  be  less  reliable  with  small  sample  sizes.  The 
normal  probability  plot  is  the  most  reliable  way  to  do  a  visual  assessment  of 
normality.  The  normal  probability  plot  compares  the  distribution  of  the  actual  data 
with  distribution  of  a  normal  distribution.  The  normal  distribution  line  is  diagonal  and 
the  plotted  data  is  compared  to  the  normal  distribution  line.  The  Shapiro-Wilks  test  is 
another  common  normality  test.  This  test  is  very  powerful  for  smaller  sample  sizes 
(~50)  but  tends  to  be  overly  sensitive  for  samples  of  over  200  (Totton  &  White, 

2011). 

Table  22  displays  the  statistics  for  skewness  produced  using  descriptive 
statistics  function  in  SPSS.  Values  above  and  below  zero  denote  non-normality  (Hair 
et  al.,  2010).  While  skewness  scores  of  zero  represent  a  perfectly  normal 
distribution,  moderately  non-normal  distributions  are  represented  by  scores  greater 
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than  2.0  (Curran,  West,  &  Finch,  1 995).  Using  a  range  for  skewness  of  ±2.0,  Table 
22  showed  nine  variables  that  exceeded  this  threshold.  In  order  to  ensure  the 
validity  of  statistical  analysis,  none  of  the  variables  were  included  in  any  constructs 
that  were  used  to  perform  regression  analysis. 

Table  23  also  displays  statistics  for  kurtosis.  Normal  distributions  display  a 
standardized  kurtosis  value  of  3.0,  and  rescaled  distributions  display  a  value  of  zero 
(Hair  et  al.,  2010).  Curran,  West,  and  Finch  (1995)  suggested  that  rescaled  kurtosis 
values  greater  than  or  equal  to  seven  are  indicative  of  moderate  non-normality.  The 
table  shows  nine  variables  that  exceeded  this  threshold.  In  order  to  ensure  the 
validity  of  statistical  analysis,  none  of  these  variables  were  included  in  any 
constructs  that  were  used  to  perform  regression  analysis. 
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Table  23.  Normality  Assessment 


Variable 

N 

Skewness 

Kurtosis 

Variable 

N 

Skewness 

Kurtosis 

Variable 

N 

Skewness 

Kurtosis 

FEAR1 

350 

-.426 

-1.113 

CPI 

350 

-.503 

.401 

TCI3  *** 

350 

7.940 

77.095 

FEAR2 

350 

-.331 

-1.155 

CP2 

350 

-.334 

.252 

TCI4 

350 

1.098 

4.757 

FEAR3 

350 

-.418 

-1.139 

CP3 

350 

-.409 

.145 

TCI  5 

350 

1.409 

5.057 

FEAR4 

351 

.133 

-1.042 

CP4 

350 

-.175 

.054 

TCI6*** 

350 

2.853 

13.738 

FEAR5 

350 

.367 

-.597 

CP5 

350 

-.339 

.487 

TCI7*** 

350 

2.737 

14.525 

FF.AR6 

350 

.369 

-.582 

CP6 

350 

-.014 

.572 

TCI8*** 

350 

2.758 

16.025 

CP7 

350 

-.266 

.095 

TCI9 

350 

.584 

.595 

CO  AUTH1 

350 

-1.089 

.571 

TCI  10 

350 

.050 

.032 

CO_AUTH2 

350 

-1.238 

1.108 

SSMA1 

350 

-1.205 

1.163 

CO_AUTH3 

350 

-1 .390 

1.909 

SSMA2 

350 

-1.395 

2.221 

D_VAL 

350 

-.084 

-1.086 

CO  AUTH4 

350 

-.089 

-1.028 

SSMA3 

350 

-.916 

.039 

CO_AUTH5 

47 

-.096 

-1.077 

SSMA4 

350 

-1.649 

2.729 

CO_AUTH6 

350 

-.506 

-.723 

SSMA5 

350 

- 1 .20 1 

1 .130 

SSMA6 

350 

-1.368 

1.529 

PALT  PI 

3  50 

.105 

-1.152 

SSMA7*** 

350 

18.447 

343.056 

PALT  P2 

350 

.303 

-1.056 

SSMA8*** 

350 

9.659 

1 15.466 

PALT  P3 

350 

-.050 

-1.201 

SSMA9*** 

350 

7.355 

62.431 

TEE  1 

47 

-1.668 

2.098 

SSMF1  + 

350 

-.042 

-.290 

TEE2 

350 

.357 

-1.074 

SSMF2* 

350 

1.027 

5.386 

TEE3 

350 

.370 

-.925 

SSMF3* 

350 

1 .446 

4.539 

TEE4 

350 

.162 

-.532 

SSMF4* 

350 

1.726 

6.957 

TEE5 

350 

-.601 

-.830 

SSMF5 

50 

-.290 

-1.869 

TEE6 

350 

.322 

-1.129 

SSMF6 

50 

.100 

-1.555 

SSMF7 

50 

-.321 

-.552 

RCI1 

47 

-1.336 

2.326 

SSMF8 

50 

-.748 

-.466 

RCI2 

350 

-1.332 

2.195 

SSMF9 

50 

.378 

-.750 

RCI3 

350 

-1.181 

1.656 

SSMF10 

50 

.181 

-1.423 

RCI4 

350 

1.198 

.730 

SSMF13 

50 

-.342 

-1 .229 

SSMF15 

50 

-.682 

-.738 

BS1 

47 

.533 

-1.084 

Cl 

350 

.757 

-.696 

BS2 

350 

-1.270 

1 .604 

C2** 

350 

1  .434 

6.889 

BS3 

350 

-1.153 

1.502 

BS4 

350 

-.636 

-.395 

TCI* 

350 

.725 

00 

BS5 

47 

-.492 

-.990 

TCI1  *** 

350 

4.720 

41.777 

TCI2*** 

350 

3.668 

38.750 

*  Transformed  using  the  square  root 
'•"•'Transformed  using  the  LOG 

♦  *  '•'Items  that  could  not  be  normalized,  and  were  not  included  in  any  tests 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-53- 


4. 


Non-Response  Bias 


The  extrapolation  method  was  used  to  estimate  bias.  This  is  the  most  commonly 
used  method  when  responses  are  received  in  “successive  waves”  (Armstrong  & 
Overton,  1977).  The  responses  were  received  in  three  successive  waves  after  follow-up 
e-mail  reminders  were  sent  to  non-respondents.  Responses  were  grouped  into  waves 
according  to  the  order  of  arrival.  Late  respondents  are  defined  as 

those  who  respond  in  the  last  wave  of  respondents  in  successive  follow¬ 
ups  to  a  questionnaire.  In  order  to  ensure  that  the  number  of  late 
respondents  is  large  enough  to  be  meaningful  practically  and  statistically, 
it  is  recommended  that  the  minimum  number  of  late  respondents  be  30. 
(Linder,  2001  p.  52) 

If  the  tests  performed  on  this  method  does 

not  produce  sufficient  statistical  evidence  to  reject  the  null  hypotheses  that 
the  means  are  equal  then  it  is  not  unreasonable  to  assert  that  the 
unobserved  group  of  non-respondents  has  similar  characteristics  to  those 
of  the  survey  respondents  and  that  non-respondents  are  missing  at 
random.  (Muir,  2010,  p.  60) 

Based  on  the  completion  date  of  the  survey,  respondents  were  grouped  into 
three  categories.  Group  one  (n  =  1 13)  represents  those  that  responded  within  the  first 
two  days  of  the  survey.  Group  two  (n  =  136)  represents  those  that  responded  after  the 
second  follow  up  e-mail  was  sent,  and  Group  three  (n  =  101)  represents  those  that 
responded  after  the  final  follow  up  e-mail  was  sent.  Multiple  analysis  of  variance 
(MANOVA)  was  then  conducted  on  nine  latent  variables  to  test  the  difference  between 
means  by  early,  middle,  and  late  responders.  Each  individual  non-response  hypothesis 
is  listed  in  Table  24. 


Table  24.  Non-Response  Bias  Hypothesis 


Variable 


Hypothesis 


Fear 


Contracting  Officer  Authority 
Planned  PALT 


Hq  —  ^Planned  PALT1  _  ^Planned  PALT2  _  ^Planned  PALT3 


Quality  of  Evaluation  Factors 
Requirement  Criticality/Importance 
Buyer  Satisfaction 
Contractor  Performance 


Hq  —  FtEEI  —  FTEE2  —  PTEE3 


Hq  —  PrcU  —  PRCI2  —  PRCI3 


Ho  —  Fbsi  —  FbS2  —  FBS3 


Source  Sel.  Method  Appropriateness  H0  =  pSSMA1  =  Fssma2  =  Pssma3 


Protest  Risk 


Hq  —  ^Protest  Riskl  —  ^Protest  Risk2  —  ^Protest  Risk3 
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Levene’s  test  was  calculated  for  each  variable.  The  results  are  listed  in  Table  25. 
Since  no  statistic  was  significant  at  a  =  0.05,  then  the  null  hypotheses  cannot  be 
rejected.  Hence,  there  is  little  evidence  that  the  variances  are  not  equal,  and 
homogeneity  of  variances  can  be  assumed  for  all  nine  variables.  Results  of  the 
univariate  F-tests  are  shown  in  Table  26.  An  examination  of  the  significance  of  the  F- 
test  statistics  reveals  that  none  are  significant  at  a  level  of  0.05.  As  a  result,  there  is  no 
statistical  evidence  to  reject  the  null  hypothesis  that  group  averages  are  equal  in  the 
nine  variables,  and  it  can  be  assumed  that  no  bias  exists  between  survey  responders 
and  survey  non-responders  in  these  cases  (Armstrong  &  Overton,  1977). 


Table  25.  Levene’s  Test 

Results 

Variable 

Levene  Statistic 

dfl 

df2 

Sig. 

FEAR 

.526 

2 

347 

.591 

CO_AUTH 

.014 

2 

347 

.986 

PALTP 

1.777 

2 

347 

.171 

TEE 

.204 

2 

347 

.815 

RCI 

2.060 

2 

347 

.129 

BS 

.686 

2 

347 

.504 

CP 

.279 

2 

347 

.757 

SSMA 

.741 

2 

347 

All 

PR 

2.526 

2 

347 

.081 
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Table  26.  Univariate  F-Tests  for  Non-Response  Bias 


Type  III  Sum  of 

Dependent  Variable  Squares 

Df 

Mean  Square 

F 

Sig. 

FEAR 

150.652a 

2 

75.326 

1.246 

.289 

COAUTH 

3.320b 

2 

1.660 

.241 

.786 

PALTP 

75.540c 

2 

37.770 

1.873 

.155 

TEE 

2.104d 

2 

1.052 

.060 

.942 

RCI 

2.474e 

2 

1.237 

.290 

.748 

BS 

CO 

CO 

CO 

CD 

2 

3.192 

1.117 

.328 

CP 

269. 81 19 

2 

134.906 

2.035 

.132 

SSMA 

4.758h 

2 

2.379 

.053 

.948 

PR 

4.769' 

2 

2.384 

.406 

.667 

Cl 

198.387j 

2 

99.193 

.957 

.385 

In  summary,  nine  variables  were  tested  for  the  presence  of  non-response  bias 
using  the  extrapolation  method  and  multivariate  analysis  of  variance.  The  results  of 
testing  did  not  provide  statistical  evidence  to  reject  the  null  hypotheses  that  the  means 
of  the  first,  second,  and  third  wave  respondents’  variables  were  equal.  It  can  be 
assumed  that  non-respondents  missing  from  the  sample  are  random  and  that  further 
analysis  will  not  be  biased  because  of  non-respondents  (Muir,  2010,  p.  60). 

E.  SAMPLE  SIZE 

To  ensure  a  valid  measure  fit  of  relationships,  reduce  any  sample  errors,  and 
reduce  the  detrimental  effects  of  non-normality,  most  researchers  suggest  using  a 
sample  size  of  200  (Hair  et  al.,  2010).  However,  it  was  also  suggested  that  a  minimum 
sample  size  of  100  could  be  used  (Smith  &  Langfield-Smith,  2004).  The  effects  of  the 
sample  size  can  be  observed  most  in  the  statistical  power  of  significance  testing,  and 
more  importantly  in  the  generalizability  of  the  results. 

According  to  Hair  et  al.  (2010),  sample  size  must  have  the  following  for  a  factor 
analysis  design: 

•  The  sample  must  have  more  observations  than  variables. 

•  The  minimum  absolute  sample  size  should  be  50  observations. 

•  Researchers  should  strive  to  maximize  the  number  of  observations  per 
variable,  with  a  desired  ratio  of  5  observations  per  variable. 
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As  a  general  rule,  the  ratio  of  observations  to  independent  variables  should  never 
be  below  5: 1 .  The  desired  amount  would  be  a  ratio  of  1 5  to  20: 1 .  If  this  ratio  can  be 
achieved,  the  results  should  be  generalizable.  This  study  contains  15  variables  with  350 
observations,  which  yields  a  ratio  of  23.33:1 .  This  ratio  more  than  satisfies  the 
referenced  generalizability  guidelines. 

F.  EXPLORATORY  FACTOR  ANALYSIS 

As  a  preliminary  step  to  establishing  construct  validity,  scales  were  purified  using 
exploratory  factor  analysis  in  SPSS  Version  22.  The  factor  loadings  matrix  produced  is 
presented  in  Table  27.  Factor  analysis  is  a  technique  whose  primary  purpose  is  to 
define  the  underlying  structure  among  variables.  It  is  primarily  used  to  analyze  the 
structure  of  the  correlations  among  a  large  number  of  variables  by  identifying  sets  of 
variables  that  have  high  influence  on  each  other,  known  as  factors  (Hair  et  al. ,  2010). 

Factor  loading  is  the  correlation  of  a  variable  and  the  factor,  while  the  squared 
loading  is  the  amount  of  the  variable’s  total  variance  accounted  for  by  the  factor  (Hair  et 
al.,  2010).  When  assessing  the  significance  of  factor  loading,  the  first  guideline  is  to  do 
a  preliminary  examination  of  the  factor  loadings.  The  general  rules  for  practical 
significance  are: 

•  factor  loadings  ranging  from  ±  .30  to  ±.40  are  the  minimal  level  for 
interpretation  of  structure, 

•  factor  loading  ±  .50  are  considered  practically  significant,  and 

•  factor  loadings  exceeding  .70  are  indicative  of  well-defined  structure. 

Interpretation  of  the  factor  matrix  starts  with  the  first  variable  of  the  first  factor  and 
moved  horizontally  from  left  to  right.  The  goal  is  to  identify  the  highest  loading  of  that 
factor.  This  is  done  for  each  variable  until  all  have  been  examined.  Variables  with 
insignificant  loadings  are  then  discarded.  When  a  variable  is  found  to  have  more  than 
one  significant  loading,  it  is  called  a  cross  loading  variable.  Cross-loaded  variables 
should  be  minimized  if  possible.  Once  all  significant  loadings  have  been  identified, 
reassessment  is  done,  removing  insignificant  and  highly  cross-loaded  variable. 

Nine  variables  were  discarded  because  of  insignificant  loadings  or  high  cross 
loadings.  Four  of  the  nine  items  were  from  established  scales.  All  other  variables 
retained  displayed  significance  factors  greater  than  the  generally  accepted  threshold  of 
.5.  Only  two  items  in  the  contracting  officer  authority  construct  contained  significance 
loadings  of  less  than  .6.  Five  items  displayed  cross-loadings  below  the  0.300  limit 
recommended  by  Hair  et  al.  (2010),  but  were  maintained  for  summation. 
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Table  27.  Factor  Loadings 


Component 

1 

2 

3 

4 

5 

6 

7 

8 

FEAR1 

.854 

FEAR2 

.885 

FEAR3 

.837 

FEAR5 

.863 

FEAR6 

.786 

COAUTH1 

.862 

COAUTH2 

.907 

COAUTH3 

.819 

COAUTH4 

.571 

.232 

COAUTH6 

.586 

-.213 

PALTP1 

.792 

PALTP2 

.827 

PALTP3 

.877 

TEE2 

.889 

TEE3 

.892 

TEE6 

.609 

RCI2 

-.906 

RCI3 

-.901 

RCI4 

-.831 

CPI 

.910 

CP2 

.928 

CP3 

.877 

CP4 

.828 

CP5 

.896 

CP6 

.807 

CP7 

.886 

SSMA1 

.866 

SSMA2 

.837 

SSMA3 

.764 

SSMA4 

.733 

SSMA5 

.762 

SSMA6 

.805 

BS3 

.264 

.734 

BS4 

.214 

.662 

BS2 

.223 

.750 

Extraction  Method:  Principal  Component  Analysis. 
Rotation  Method:  Oblimin  with  Kaiser  Normalization, 
a.  Rotation  converged  in  6  iterations. 
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Summated  scales  combine  several  like  variables  into  a  single  composite 
measure.  This  is  done  by  combining  the  variables  that  are  highly  loaded  on  a  factor. 
Summated  scales  have  two  specific  benefits:  They  provide  a  means  of  overcoming  to 
some  extent  the  measurement  error  inherent  in  all  measured  variables  and  its  ability  to 
represent  multiple  aspects  of  a  concept  in  a  single  measure  (Hair  et  al.,  2010). 
Descriptions  for  the  summated  scales  can  be  seen  in  Table  28. 
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Table  28.  Construct  Descriptives 


Summated  Construct 

N  Min 

Max 

Mean 

Skewnes 

Std.  Deviations 

Kurtosis 

FEAR 

350  5 

35 

19.71 

7.782 

-.107 

-.798 

Contracting  Officer  Authority  (CO_AUTH) 

350  5 

35 

25.05 

6.021 

-.820 

.447 

PALT  Planned  (PALT_P) 

350  3 

20 

10.81 

4.502 

-.092 

-.921 

Tech  Eval  Effectiveness  (TEE) 

350  3 

21 

10.34 

4.177 

.315 

-.641 

Requirement  Criticality  (RCI) 

350  4 

21 

17.15 

3.008 

-.847 

.856 

Buyer  Satisfaction  (BS) 

350  6 

21 

13.99 

1.691 

-.483 

2.861 

Contractor  Performance  (CP) 

350  7 

49 

33.08 

8.166 

-.201 

.282 

Source  Sel.  Method  Appropriateness 
(SSMA) 

350  6 

42 

33.06 

6.655 

-1.174 

1.585 
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Reliability  is  also  a  consideration  in  assessing  the  degree  of  consistency  between 
multiple  measures  of  a  variable  (Hair  et  al.,  2010).  Reliability  is  a  necessary,  but  not 
sufficient,  aspect  of  construct  validity.  One  measure  of  construct  reliability  is  the 
Cronbach’s  Alpha,  which  is  also  known  as  the  reliability  coefficient.  The  most  widely 
used  lower  limit  for  Cronbach’s  Alpha  is  .70,  although  .60  may  be  acceptable  in 
exploratory  research  (Nunnally,  1978).  Table  29  shows  the  reliability  coefficients  for 
summated  scales  resulting  from  exploratory  factor  analysis. 

Table  29.  Cronbach’s  Alpha  Scores 


Construct 

Cronbach’s 

Alpha 

FEAR 

.904 

CO_AUTH 

.826 

PALT_P 

.805 

TEE 

.758 

RCI 

.856 

BS 

.880 

CP 

.963 

SSMA 

.901 

G.  HYPOTHESIZED  MODELS 

The  final  step  of  analysis  was  the  regression  analysis  of  the  hypothesized 
regression  models  in  SPSS  version  22.  Three  methods  of  regression  were  used  to 
analyze  various  hypothesized  relationships  between  constructs.  Multiple  regression  was 
used  to  measure  the  left  side  of  the  model,  which  analyzed  the  factors  that  contributed 
to  protest  fear.  Simple  linear  and  logistic  regressions  were  performed  on  the  right  side 
of  the  model  to  measure  the  resultant  effects  of  protest  fear. 

The  model  summary  of  each  regression  featuring  the  R2  and  adjusted  R2  values 
was  analyzed  to  assess  the  total  variation  explained  in  the  dependent  variable  by  the 
independent  variable(s).  The  significance  value  in  the  Analysis  of  Variation  (ANOVA) 
table  was  then  examined  to  determine  the  overall  validity  of  the  model.  To  test  the 
validity  of  the  model,  the  following  hypotheses  are  assumed: 

•  H0:  Pi  =  (B2  =  •••  =  Pk  =  0 
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•  Hx:  At  least  one  Pi  is  not  equal  to  0 

If  the  null  hypothesis  is  true,  none  of  the  independent  variable  is  linearly  related 
to  the  dependent  variable,  and  the  model  is  invalid  or  not  statistically  significant.  A 
minimum  of  one  px  must  be  not  equal  to  0  for  the  model  to  have  validity.  The 
significance  test  in  the  ANOVA  table  combines  multiple  f-tests  into  a  single  test  (Keller, 
2008). 

The  Coefficients  table  is  the  last  part  of  the  regression  analysis.  The 
Unstandardized  Coefficients  (P)  describe  the  relationship  between  each  independent 
variable  and  the  dependent  variable.  The  significance  value  for  each  independent 
variable  was  then  analyzed  to  determine  statistical  significance  of  each  independent 
variable.  All  tests  used  an  alpha  (a)  value  of  0.05,  meaning  that  any  significance  value 
greater  than  a  was  deemed  not  significant.  Alpha  values  between  0.05  and  0.10  were 
considered  marginally  significant. 

1 .  Antecedents  of  Fear  Results 

Multiple  regression  analysis  was  performed  on  the  summated  constructs  that 
measured  the  relationships  identified  in  the  left  side  of  the  model.  A  visual  depiction  of 
the  model  is  shown  in  Figure  2.  The  model  used  Planned  PALT  (PALT_P),  Dollar  Value 
(D_VAL),  Competency  (Cl  and  C2),  Requirement  Criticality/Importance  (RCI),  and 
Protest  Risk  (PR)  as  independent  variables  measuring  the  variance  in  the  dependent 
variable  Fear  (FEAR). 

The  statistical  significance  of  the  model  was  then  examined  in  the  ANOVA  table. 
The  significance  of  the  model  was  reported  at  .000  using  an  a  value  of  0.05.  Since  the 
significance  is  less  than  a,  we  can  reject  the  null  hypothesis  that  there  is  no  linear 
relationship  between  FEAR  and  the  independent  variables  in  the  model.  The  ANOVA 
table  is  shown  in  Table  31 . 

The  Model  Summary  table  for  this  model  can  be  seen  in  Table  30.  The 
Correlation  Coefficient  (R)  was  .361 .  The  Coefficient  of  Determination  (R2)  was  .131 
with  an  adjusted  R2  of  .1 15.  The  R2  value  means  that  13.1  percent  of  the  variation  in  the 
dependent  variable  is  explained  by  the  independent  variables.  The  proportion  left 
unexplained,  which  is  called  the  coefficient  of  alienation,  is  .869.  Sirkin  (2006,  p.  474) 
noted  that  there  are  “many  real  world  factors  that  mitigate  linearity  in  social  sciences.  It 
would  be  rare  indeed  to  see  coefficients  of  determination  approaching  1 .0  (or  even  .9 
or.8)  in  social  or  behavioral  research.”  He  then  added  that  “R2  is  a  useful  concept  in 
statistics,  it  is  less  useful  in  the  real  world  of  data  analysis  in  determining  what  is  or  is 
not  a  “good”  linear  relationship.  Thus  statistical  significance  is  often  used  to 
demonstrate  “good”  relationships”  (Sirken,  2006,  p.  457).  While  the  R2  values  are  high 
enough  to  substantiate  this  model,  the  model  also  suggests  that  there  are  some 
unobserved  factors  that  may  explain  more  of  the  variance  in  the  FEAR  of  protest 
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variable. 


Finally,  the  Coefficients  Table  was  examined  for  the  (3  values  and  significance  of  each 
independent  variable.  Five  out  of  six  independent  variables  had  significance  values  less  than  .05, 
with  the  variable  Years  of  Experience  having  a  significance  value  of  .379.  PALT  Planned  had  a 
(3  value  of  -.259  and  a  significance  of  .004.  This  shows  that  as  PALT  is  increased,  the  level  of 
FEAR  decreases  by  .259  (on  a  seven-point  scale)  assuming  all  other  variables  remain  constant. 
Dollar  Value  had  a  P  value  of  .72 1  with  a  significance  value  of  .003.  This  shows  that  as  Dollar 
Value  increases  by  one  unit,  FEAR  also  increases  by  .721  (on  a  seven-point  scale)  assuming  all 
other  variables  remain  constant.  It  is  important  to  note  that  the  Dollar  Value  construct  does  not 
measure  the  actual  dollar  value  of  the  acquisition.  The  actual  dollar  values  were  not  normally 
distributed  and  therefore  inappropriate  for  regression  analysis.  Requirement  Criticality  had  a  P 
value  of  .43 1  with  a  significance  value  of  .002.  This  shows  that  as  Requirement  Criticality 
increases  by  one  unit,  FEAR  also  increases  by  .43 1  (on  a  seven-point  scale)  assuming  all  other 
variables  remain  constant.  Protest  Risk  had  a  P  value  of  .374  with  a  significance  value  of  .029. 
This  shows  that  as  Protest  Risk  increases  by  one  unit,  FEAR  also  increases  by  .374  (on  a  seven- 
point  scale)  assuming  all  other  variables  remain  constant.  Years  of  Experience  had  a  P  value  of 
.037  but  was  insignificant.  The  number  of  Source  Selections  had  a  P  value  of  -.267  with  a 
significance  value  of  .01 1.  This  shows  that  as  the  Number  of  Source  Selections  increases  by  one 
unit,  FEAR  decreases  by  .267  (on  a  seven-point  scale)  assuming  all  other  variables  remain 
constant.  Results  of  this  test  can  be  found  in  Table  32.  A  visual  depiction  of  all  relationships  is 
shown  in  Figure  2. 


Ta 

ble  30.  Model  Summary 

Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.36 1 a 

.131 

.115 

7.319 

a.  Predictors:  (Constant),  Number  of  Source  Selections,  Protest 
Risk,  Dollar  Value,  Requirement  Criticality,  PALT  Planned,  Years  of 
Experience 


Tab 

e  31.  ANOVAa 

Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

2759.668 

6 

459.945 

8.587 

,000b 
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Residual 

18373.021 

343 

53.566 

Total 

21132.689 

349 

a.  Dependent  Variable:  FEAR 

b.  Predictors:  (Constant),  Number  of  Source  Selections,  Protest  Risk,  Dollar  Value, 
Requirement  Criticality,  PALT  Planned,  Years  of  Experience 


Table  32.  Coefficients 


Unstandardized 

Coefficients 

Standardized 

Coefficients 

Construct 

B 

Std.  Error 

Beta 

t 

Siq. 

(Constant) 

10.995 

2.852 

3.856 

.000 

PALT  Planned 

-.259 

.091 

-.150 

-2.862 

.004 

Dollar  Value 

.721 

.238 

.155 

3.032 

.003 

Requirement  Criticality 

.431 

.135 

.167 

3.183 

.002 

Protest  Risk 

.374 

.171 

.116 

2.190 

.029 

Years  of  Experience 

.037 

.042 

.049 

.881 

.379 

Number  of  Source 

Selections 

-.267 

.104 

-.142 

-2.558 

.011 

a.  Dependent  Variable:  FEAR 
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2.  Consequences  of  Fear  Results 

One  of  the  objectives  of  this  research  is  to  determine  the  consequences  of 
protest  fear.  Multiple  variables  were  used  to  gauge  the  lengths  that  contracting 
personnel  would  take  to  avoid  a  protest.  Some  constructs,  such  as  the  amount  of 
awarded  contracts,  were  very  straightforward  in  ascertaining  whether  a  problem  exists. 

The  right  side  of  the  model  was  then  tested  through  a  combination  of  simple 
linear  regressions  and  logistic  regressions  in  SPSS  Version  22.  Key  statistics  of  these 
tests  can  be  found  in  Table  33.  Logistic  regression,  being  well  suited  for  analyzing 
dichotomous  outcomes,  is  finding  increased  popularity  in  social  science  research  (Peng 
&  So,  2002).  It  is  the  appropriate  statistical  technique  for  performing  analysis  when  the 
dependent  variable  is  categorical.  Hair  et  al.  (2011)  stated  that  Logistic  Regression  is 
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best  suited  to  deal  with  two  research  objectives:  identifying  the  independent  variables 
that  impact  group  membership  in  the  dependent  variable,  and  establishing  classification 
systems  based  on  the  logistic  model  for  determining  group  membership.  Logistic 
regression  estimates  the  probability  that  a  particular  outcome  will  occur.  The 
relationship  between  independent  and  dependent  variable  is  expressed  in  the  form  of 
an  odds  ratio,  which  expresses  the  probability.  If  the  ratio  is  positive,  “an  increase  in  that 
independent  variable  will  result  in  an  increase  in  the  probability  of  the  event”  (Keller, 
2008,  p.  742). 

Linear  regression  was  performed  to  analyze  the  relationship  between  the  FEAR 
of  protest  and  technical  evaluation  effectiveness  (TEE),  with  FEAR  being  the 
independent  variable  and  TEE  being  the  dependent  variable.  TEE  was  measured  using 
reverse-coded  survey  questions  (items);  thus,  the  inverse  of  the  given  responses  was 
used  in  the  test.  There  was  a  significant  relationship  (.000)  between  the  two  variables. 
The  (3  coefficient  was  -.136,  which  was  in  the  hypothesized  direction. 

The  second  test  performed  was  a  linear  regression  analyzing  the  relationship 
between  FEAR  of  protest  and  perceived  source  selection  method  appropriateness 
(SSMA),  using  FEAR  as  independent  variable  and  SSMA  as  dependent  variable.  The 
test  failed  to  reject  the  null  hypothesis  (.275)  that  there  was  no  linear  relationship 
between  FEAR  and  SSMA. 

Since  this  analysis  dealt  with  the  consequences  of  protest  fear,  some  aspects  of 
different  constructs  were  transformed  into  binary-coded  variables.  SSMF1  and  SSMF2 
were  used  to  measure  the  difference  between  planned  and  actual  procurement  lead- 
time  (PALT).  SSMF1  asked  how  many  days  were  planned  for  the  Source  Selection  in 
question,  and  SSMF2  asked  how  many  days  the  Source  Selection  actually  consumed. 
To  understand  the  effects  of  the  independent  variable  FEAR  of  protest  on  Actual  PALT, 
source  selections  that  took  longer  than  planned  were  coded  as  1  and  those  that  were 
less  or  the  same  were  coded  as  0.  This  model  was  significant  (.000),  but  the  Odds  Ratio 
of  1 .051  indicates  slightly  higher  than  1  to  1  odds  of  having  increased  Actual  PALT  due 
to  protest  fear. 

SSMF3  and  SSMF4  were  used  to  measure  resources  consumed  to  conduct  the 
source  selection.  SSMF3  asked  how  many  personnel  were  on  the  Source  Selection 
team  and  SSMF4  asked  how  many  there  should  have  been  absent  a  fear  of  protest. 
Three  tests  were  run  using  FEAR  of  protest  as  the  independent  variable.  The  first  was  a 
linear  regression  using  SSMF3  as  the  dependent  variable.  This  test  was  significant 
(.000)  with  a  (3  coefficient  of  .010.  Thus,  as  the  fear  of  protest  increases,  so  does  the 
number  of  personnel  assigned  to  a  source  selection.  The  next  test  used  a  binary 
variable  for  FEAR  of  protest  labeled  FEAR  HI/LO,  where  a  respondent  whose 
summated  FEAR  rating  was  above  average  was  coded  as  1 ,  and  all  others  were  coded 
as  0.  SSMF3  again  was  the  dependent  variable.  Again,  a  linear  regression  was 
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performed  since  the  dependent  variable  was  not  binary.  This  model  was  also  significant 
(.000),  but  displayed  a  higher  (3  coefficienct  of  .149.  The  last  test  was  a  logistic 
regression  using  the  difference  between  SSMF3and  SSMF4.  Those  responses  that  had 
more  people  on  the  team  than  the  respondent  thought  should  be  (absent  a  fear  of 
protest)  were  coded  as  1 ;  all  others  were  coded  as  0.  This  test  was  not  significant 
(.379). 

A  series  of  logistic  regressions  were  then  run  on  the  appropriateness  of  two 
types  of  source  selections — LPTA  and  Full  Trade-Off — using  responses  from  the 
variables  SSMF6  and  SSMF8.  The  first  step  in  the  test  was  to  filter  in  only  the 
respondents  that  used  the  particular  Source  Selection  Method  (i.e.,  LPTA  or  full  trade¬ 
off).  A  dummy  variable  was  then  created  to  ascertain  how  appropriate  the  respondent 
thought  the  use  of  the  method  was.  The  questions  used  a  seven-point  Likert-type  scale 
with  1  being  “Completely  Inappropriate”  and  7  being  “Completely  Appropriate.” 
Respondents  that  answered  the  questions  with  scores  of  3  or  less  were  coded  as  1,  and 
all  others  were  coded  as  0.  Of  note,  there  were  13  respondents  out  of  133  that  used 
LPTA  even  though  they  felt  it  was  to  some  degree  inappropriate.  10  respondents  out  of 
174  that  used  Full  Trade-off  even  though  they  felt  it  was  to  some  degree  inappropriate. 
This  inappropriate  use  totals  7.49  percent  of  the  307  respondents  that  used  either  LPTA 
or  trade-off  as  a  source  selection  method.  Both  tests  used  the  same  binary  FEAR  HI/LO 
independent  variable  in  the  aforementioned  tests.  The  logistic  regression  test  to 
measure  the  relationship  with  the  use  of  the  Full  Trade-off  inappropriateness  was  not 
significant  (.181).  The  test  using  LPTA  inappropriateness  was  significant  (.010)  with  an 
Odds  Ratio  of  4.673. 

Three  tests  were  then  performed  to  measure  the  relationship  between  the  FEAR 
of  protest  and  the  use  of  discussions  using  SSMF9,  SSMF10,  and  SSMF12.  The  first  test 
used  the  binary  FEAR  HI/LO  independent  variable  to  measure  the  level  of  satisfaction 
with  the  degree  of  discussions,  using  SSMF9  as  the  dependent  variable.  SSMF9  uses  a 
seven-point  Likert-type  with  1  being  “Completely  Satisfied”  and  7  being  “Completely 
Dissatisfied.”  The  test  was  significant  (.037)  with  a  (3  coefficient  of  .382.  The  test 
signifies  that  if  FEAR  of  protest  is  high,  the  level  of  dissatisfaction  with  discussions  rises 
slightly.  The  next  test  used  the  FEAR  HI/LO  binary  independent  variable  and  SSMF10  as 
the  dependent  variable.  SSMF10  examined  how  much  the  level  of  discussions  would 
change  if  there  were  no  ability  to  protest.  It  used  a  seven-point  Likert-type  scale  with  1 
representing  “No  Difference”  and  7  representing  a  “Substantial  Difference.”  This  test 
was  also  significant  (.000)  with  a  (3  of  1 .289.  The  test  signifies  that  respondents  with 
high  FEAR  of  protest  would  change  the  manner  in  which  they  conducted  discussions  if 
there  were  no  threat  of  protest.  The  last  test  was  a  Logistic  Regression  performed  on 
FEAR  as  the  independent  variable  and  SSMF12.  SSMF12  asked  whether  discussions 
had  been  conducted  or  not,  and  was  binary  coded  where  1  represented  Yes  and  0 
represented  No.  This  test  was  also  significant  (.037),  but  the  Odds  Ratio  of  1 .030  shows 
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only  a  slightly  better  than  to  1  odds  of  FEAR  having  an  effect  on  whether  discussions 
are  conducted.  Hence,  fear  of  protest  tends  to  slightly  increase  the  odds  that 
discussions  will  occur. 

The  next  test  was  a  linear  regression  performed  to  analyze  the  appropriateness 
of  oral  presentations.  The  test  used  FEAR  as  an  independent  variable  and  SSMF15  as 
the  dependent  variable.  This  test  did  not  reject  the  null  hypothesis  (.261 )  that  there  is  no 
linear  relationship  between  the  two  variables. 

The  Awarded  Contracts  variable  (AC)  measured  the  difference  between  how 
many  contracts  were  planned  and  how  many  were  actually  awarded.  The  AC  variable 
was  binary  coded  where  respondents  that  saw  more  contracts  awarded  than  planned 
was  coded  as  1  and  all  others  were  0.  There  were  19  instances  where  more  contracts 
were  awarded  than  originally  planned,  representing  5.42  percent  of  respondents. 
However,  this  test  was  insignificant  (.350)  and  no  further  analysis  of  this  relationship 
was  done. 

Transaction  Costs  consists  of  two  separate  variables  comprised  of  questions 
measuring  the  number  of  iterations  of  key  source  selection  documents  (TCI)  and 
personnel  costs  of  those  associated  with  the  source  selection  (TCP).  Descriptive 
statistics  of  the  TCI  variables  are  shown  in  Table  33.  TCP  was  calculated  by  asking 
respondents  the  number,  rank,  and  fraction  of  one  year’s  time  spent  on  the  source 
selection  by  role.  The  average  2013  General  Schedule  annual  pay  was  used  to 
calculate  the  cost  for  each  role,  and  then  summed  for  each  source  selection  into  the 
TCP  dependent  variable.  The  values  ranged  from  a  low  of  $7,000.00  to  a  maximum  of 
$3,551 ,944.33.  The  average  cost  per  source  selection  was  $235,236.34,  with  a 
standard  deviation  of  $291 ,620.05.  Linear  regression  was  used  to  test  the  relationship 
between  the  independent  variable,  FEAR  of  protest,  and  TCP.  The  test  was  significant 
(.018)  but  displayed  a  very  low  (3  of  .009.  While  there  was  a  statistically  significant 
relationship  between  these  two  variables,  the  effect  size  is  minute. 

The  next  series  of  tests  were  on  the  TCI  variables.  Normality  was  achieved  on 
only  four  types  of  source  selection  documents  of  many  types  of  documents  measured — 
Source  Selection  Plan,  Debriefing  Script,  Color/Rating  Chart,  and  Evaluation/Briefing 
Chart  to  the  Source  Selection  Authority.  The  first  test  measured  the  relationship 
between  the  independent  variable,  FEAR,  and  TCI4,  which  represents  the  number  of 
iterations  of  the  Source  Selection  Plan.  This  test  was  shown  to  be  significant  (.000),  and 
had  a  p  of  .023.  The  second  test  measured  the  relationship  between  the  independent 
variable,  FEAR,  and  TCI5,  which  represents  the  number  of  iterations  of  the  Debriefing 
Script.  This  test  was  also  shown  to  be  significant  (.000),  and  had  a  p  of  .028.  The  third 
test  measured  the  relationship  between  the  independent  variable,  FEAR,  and  TCI9, 
which  represents  the  number  of  iterations  of  the  Color/Rating  Chart.  This  test  was 
shown  to  be  significant  (.000),  and  had  a  p  of  .020.  Finally,  the  last  test  measured  the 
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relationship  between  the  independent  variable,  FEAR,  and  TCI10,  which  represents  the 
number  of  iterations  of  the  Evaluation/Briefing  Chart  to  the  Source  Selection  Authority. 
This  test  was  shown  to  be  significant  (.000),  and  had  a  (3  of  .018.  While  there  is  a 
significant  relationship  between  FEAR  of  protest  and  the  number  of  changes  to  each  of 
these  documents,  the  magnitudes  of  the  relationships  were  small. 


Table  33.  Document  Iteration  Descriptive 


Document 

N 

Min 

Max 

Mean 

Std.  Deviation 

Source  selection  decision  document 

350 

.00 

12.00 

2.6429 

1.80604 

Comp,  assessment/proposal  analysis  report 

350 

.00 

10.00 

2.4743 

1.66307 

Evaluation  notices 

350 

.00 

21.00 

2.3314 

2.55796 

Source  selection  plan 

350 

.00 

12.00 

2.7514 

2.02093 

Debriefing  scripts 

350 

.00 

15.00 

1.9657 

1.98099 

Technical  evaluations 

350 

.00 

17.00 

3.4143 

2.38244 

Past  performance  evaluations 

350 

.00 

20.00 

2.7486 

2.62666 

Cost/Price  analysis 

350 

.00 

17.00 

2.3086 

1.87937 

Color/rating  chart 

350 

.00 

11.00 

1.3257 

1.60128 

Evaluation  briefing  charts  for  reviewers  and  SSA 

350 

.00 

10.00 

1.7257 

1.61624 

The  last  hypothesized  consequence  of  FEAR  to  be  tested  was  the  effect  on 
contracting  officer  authority.  Two  tests  were  performed  using  CO  AUTH  as  the 
dependent  variable.  The  first  test  used  the  binary  FEAR  HI/LO  and  the  second  test  used 
a  continuous  measure  of  FEAR  (on  a  one  to  seven  scale)  as  the  independent  variable. 
The  first  test  had  a  significance  of  .914,  which  is  not  significant.  The  second  test  had  a 
significance  of  .093  which  is  considered  marginally  significant  using  an  a  value  of  .10. 
The  (3  coefficient  was  -.070,  which  was  in  the  direction  hypothesized. 

The  object  of  the  final  set  of  tests  was  to  analyze  hypothesized  relationships 
between  the  technical  evaluations’  effectiveness  (TEE)  and  perceived  source  selection 
method  appropriateness  on  contractor  performance  (CP).  Multiple  regression  was  used 
to  analyze  this  relationship.  The  construct  TEE  and  SSMA  were  used  as  the 
independent  variables  and  CP  was  used  as  the  dependent  variable.  The  overall  test 
was  significant  (.000).  TEE  was  significant  as  an  independent  variable  (.001 )  and  had  a 
P  of  .327.  SSMA  was  also  significant  as  an  independent  variable  (.000)  and  had  a  p  of 
.328.  This  test  showed  significant  relationships  between  the  TEE  and  CP,  and  between 
SSMA  and  CP.  The  ps  indicated  moderate  effect  sizes. 
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Next,  two  exploratory  tests  were  performed  on  the  inappropriate  use  of  LPTA  as 
the  independent  variable  and  contractor  performance  as  the  dependent  variable.  Only 
the  133  responses  that  used  LPTA  were  included  in  these  tests.  For  the  first  test,  a 
binary  variable  was  used,  in  which  respondents  that  felt  LPTA  was  anything  other  than 
appropriate  (4  or  less)  were  coded  as  1  and  all  others  were  0.  The  test  showed  a 
significant  relationship  (.010)  between  the  variables,  and  a  (3  of  -4.355  indicating  that  CP 
decreases  significantly  for  those  respondents  that  felt  the  use  of  LPTA  was  other  than 
appropriate.  The  second  test  used  the  continuous  LPTA  appropriateness  scores  where 
1  represents  “completely  inappropriate”  and  7  represents  “completely  appropriate.”  CP 
was  the  dependent  variable.  This  test  was  also  significant  (.006)  with  a  (3  of  1 .237.  This 
shows  that  as  LPTA  appropriateness  increases  by  one  unit,  CP  also  increases  by  1 .237 
(on  a  seven-point  scale). 

The  final  test  was  a  linear  regression  to  measure  the  relationship  between 
contractor  performance  and  buyer  satisfaction.  The  construct  CP  was  used  as  the 
independent  variable  and  BS  was  used  as  the  dependent  variable.  The  test  showed  a 
significant  (.000)  positive  relationship  between  the  two  variables;  the  magnitude  of  the  (3 
was. 285.  A  summary  table  of  all  key  regression  data  can  be  found  in  Table  34.  A  visual 
depiction  of  all  relationships  associated  with  the  right  side  of  the  model  can  be  found  in 
Figure  3. 

Table  34.  Key  Statistics  from  Right-Side  (Consequences)  Regressions 
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Independent 

Variable 

Dependent  Variable 

N 

R2 

AdiR2 

D.F. 

Sig. 

P 

P  Sig. 

FEAR 

TEE 

350 

.064 

.061 

349 

.000 

-.136 

.000 

FEAR 

PALT  Actual" 

350 

.047 

- 

349 

.000 

1  051' 

.001 

FEAR 

SSMA 

350 

.003 

.001 

349 

.275 

-.050 

.275 

FEAR  HI/LO 

SSMFe"(LPTA  Appropriateness) 

133 

.048 

- 

132 

010 

4673' 

.015 

FEAR  HI/LO 

SSMF7"(T/0  Appropriateness) 

174 

.010 

- 

173 

.181 

.416' 

.187 

FEAR  HI/LO 

SSMF9  (Satisfaction  w /  discussions) 

350 

.012 

010 

349 

037 

.382 

037 

FEAR  HI/LO 

SSMF'0  (Hypothetical  change) 

350 

.100 

097 

349 

.000 

1.289 

000 

FEAR 

SSMF'2"  (Discussions  Held  Y/N) 

350 

.012 

- 

349 

.037 

1  030' 

.038 

FEAR 

SSMF'5  (Oral  Pres  Appropriateness)  350 

.004 

.001 

349 

.261 

-.016 

.261 

FEAR 

SSMF3  (Resources) 

350 

.077 

.074 

349 

000 

.010 

.000 

FEAR  HI/LO 

SSMF3  (Resources) 

350 

.066 

063 

349 

.000 

.149 

.000 

SSMF3"  (More  Resources  than 

349 

FEAR 

needed) 

350 

.002 

- 

.378 

1013* 

.379 

FEAR 

TCI 

350 

.101 

.099 

349 

000 

.014 

.000 

FEAR 

TCI4  (Source  Selection  Plan) 

350 

.084 

082 

349 

.000 

.023 

.000 

FEAR 

TCI5  (Debriefing  Scripts) 

350 

.078 

076 

349 

.000 

.028 

.000 

FEAR 

TCI9  (Color/Rating  Chart) 

350 

038 

035 

349 

.000 

020 

000 

TCI 10  (Evaluation/Briefing  Chart  for 

349 

FEAR 

SSA) 

350 

.038 

035 

.000 

018 

.000 

FEAR 

TCP 

270 

.021 

017 

269 

018 

.009 

.018 

FEAR 

AC" 

350 

.002 

- 

349 

.350 

1.029' 

.355 

FEAR 

CO  AUTH 

350 

.008 

005 

349 

.093 

-.070 

.093 

FEAR  HI/LO 

CO  AUTH 

350 

.000 

-.003 

349 

914 

-070 

.914 

TEE" 

CP 

350 

.123 

.118 

349 

.000 

.327 

.001 

SSMA" 

CP 

.328 

.000 

SSMFe"(LPTA 

132 

Appr) 

CP 

133 

.049 

.042 

010 

-4.355 

.010 

SSMFe 

CP 

133 

.057 

049 

132 

.006 

1  237 

.006 

CP 

BS 

350 

.442 

.441 

349 

.000 

.285 

.000 

Exp  (3  from  Logistic  Regression 
Binary 

Multiple  Regression 
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Figure  3.  Right  Side  of  Hypothesized  Model 
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fRAtSIANTIA  rER  SCIENTlAu 
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H.  CONCLUSION 


In  summary,  this  chapter  reported  the  results  of  the  linear,  multiple,  and 
logistic  regressions  performed  on  the  350  usable  survey  responses.  Firstly,  the 
350  responses  were  analyzed  to  verify  that  the  data  was  appropriate  for  being 
used  in  regression  analysis  (i.e.,  that  the  assumptions  of  regression  were 
satisfied).  Exploratory  Factor  Analysis  was  then  performed  to  purify  scales,  in 
which  35  items  were  retained  for  eight  constructs,  with  all  constructs  being 
measured  by  no  fewer  than  three  items  and  exhibiting  sufficient  reliability 
coefficients  (Muir,  2010).  Finally,  the  hypothesized  model  was  analyzed  using 
regression  analysis.  Several  relationships  were  shown  to  be  statistically 
significant,  which  supported  12  of  14  proposed  hypotheses.  Lastly,  the 
hypothesized  model  was  presented  with  only  the  statistically  significant 
relationships  shown  (Figure  4).  For  ease  of  interpretation,  a  summary  table 
containing  the  significance  level  of  each  hypothesis  can  be  found  in  Table  35. 
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Hypothesis 

HI 

H2 

H3 

H4 

H5 

H6 

H7 

H8 

H9 

H10 

H1 1 

H12 

H13 

H14 


_ Table  35.  Summary  of  Hypothesis  Testing _ 

_ Supported? 

Insufficient  PALT  has  a  direct  positive  relationship  on  the  level  of  fear  of  protest.  Yes 
The  greater  a  contracting  officer’s  Competence  level,  the  lower  the  level  of  fear  Yes 

of  protest. 

The  level  of  requirement  criticality/importance  has  a  positive  impact  on  the  level  Yes 
of  fear  of  protest. 

As  protest  risk  increases,  the  level  of  fear  of  protest  increases.  Yes 

The  level  of  fear  of  protest  has  a  negative  effect  on  the  technical  evaluation  Yes 

effectiveness. 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  Source  Partially 

Selection  method  fit/  appropriateness. 

There  is  a  positive  correlation  between  the  fit/appropriateness  of  a  source  Partially 

selection  and  contractor  performance. 

There  is  a  positive  relationship  between  technical  evaluation  effectiveness  and  Yes 
contractor  performance. 

There  is  a  direct  positive  relationship  between  contractor’s  performance  based  Yes 
on  the  source  selection  method  used  and  the  Buyer’s  satisfaction. 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  actual  Yes 

PALT. 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  the  number  of  No 
Human  Resources  assigned  to  the  source  selection  team. 

There  is  a  direct  positive  relationship  between  fear  of  protest  and  transaction  Yes 

costs. 

The  higher  the  level  of  fear  of  protest,  the  greater  the  number  of  awarded  No 

contracts. 


There  is  a  direct  negative  relationship  between  fear  of  protest  and  the  contracting  Yes 
officer’s  Authority. 
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V.  DISCUSSION  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

The  objective  of  this  research  was  to  address  gaps  in  the  literature  review 
and  offer  acquisition  leaders,  practitioners,  and  scholars  a  better  understanding  of 
the  causes  and  effects  of  a  fear  of  protest.  From  this  research,  acquisition  leaders 
and  practitioners  can  better  align  their  internal/external  policies,  acquisition 
strategies,  and  resources  to  mitigate  the  causes  and  effects  of  fear  of  protest,  which 
ultimately  saves  money  and  increases  customer  satisfaction  (mission)  and 
contractor  performance. 

The  existing  literature  (Manuel  &  Schwartz,  2011;  Schwartz  et  al. ,  2013; 
O’Rourke,  2014),  Gordon’s  publication  (2013),  and  other  publications  (Knauth,  2013; 
Maser  et  al.,  2010;  Kendall,  2012),  were  used  to  identify  problems  that  have  not 
been  addressed,  specifically,  the  magnitude  of  fear  of  protests,  the  extent  that  fear 
of  a  protest  affects  an  acquisition  strategy,  and  the  lengths  that  acquisition 
professionals  would  go  to  avoid  a  protest.  These  problems,  to  this  point,  have  been 
largely  anecdotal.  This  study  is  the  first  to  assess  the  magnitude  of  fear  of  protest, 
and  the  first  to  empirically  explore  its  antecedents  and  consequence. 

To  close  the  gaps  in  the  existing  literature,  this  research  addressed  the 
following  three  research  questions: 

Research  Question  1:  Are  contracting  officers  sufficiently  concerned 
about  bid  protests  to  alter  acquisition  strategies? 

Research  Question  2:  If  so,  what  factors  affect  protest  “fear”? 

Research  Question  3:  What  are  the  consequences  to  the  acquisition 
strategy  and  contract  outcomes? 

Because  of  the  nature  of  the  research  questions,  it  was  appropriate  to  use  a 
combination  of  qualitative  and  quantitative  methodologies.  Literature  from  several 
governmental,  public,  and  private  studies  and  reports  were  used  to  develop  the 
conceptual  model,  14  hypotheses,  an  interview  questionnaire,  and  an  online  1 15- 
question  survey.  Once  the  data  was  collected  and  analyzed  from  350  responses  that 
represented  a  diverse  population  of  U.S.  Navy  contracting  officers,  regression 
analysis  was  performed  using  SPSS  Statistics  Version  22.0. 

From  the  statistical  analysis,  many  of  the  hypotheses  had  significant 
relationships.  Overall,  the  magnitude  of  those  relationships  was  relatively  small. 
Among  the  more  significant  outcomes,  the  research  determined  that  fear  of  protest 
decreases  the  technical  evaluation  effectiveness,  which  decreases  the  contractor’s 
performance,  and  ultimately  decreases  buyer  satisfaction.  Additionally,  fear  of  a 
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protest  is  linked  to  inappropriate  use  of  the  LPTA  source  selection  method,  to  delays 
in  contract  awards,  to  increased  transaction  costs,  and  to  dissatisfaction  with 
discussions  with  offerors. 

B.  DISCUSSION  AND  IMPLICATIONS 

In  this  section,  the  research  results  as  well  as  managerial  implications  are 
discussed.  Additionally,  the  results  of  the  hypothesis  testing  and  regression  analysis 
are  discussed.  The  analysis  resulted  in  statistically  significant  estimates  for  10  of  the 
hypotheses  (Hypotheses  1-10  and  12),  marginally  significant  estimates  for  one  of 
the  hypotheses  (14),  and  estimates  with  no  statistical  support  for  two  of  the 
hypotheses  (1 1  and  13) 

This  section  is  organized  as  follows.  First,  the  question  of  what  is  the 
magnitude  of  fear  of  protest  among  U.S.  Navy  contracting  officers  was  answered 
and  discussed.  Secondly,  each  result  of  hypothesis  testing  is  discussed  and 
explained. 

1.  Magnitude  of  Fear 

A  histogram  of  the  values  for  fear  of  protest  can  be  found  below  in  Figure  5. 
The  left  side  indicates  less  fear  and  the  right  indicates  more.  For  clarification,  16 
respondents  indicated  that  they  strongly  disagreed,  indicating  a  low  level  of  fear,  and 
eight  indicated  that  they  strongly  agreed,  indicating  a  high  level  of  fear.  The 
combined  average  response  that  indicates  neither  agree  nor  disagree  is  19.71  on  a 
possible  scale  range  from  five  to  35.  This  would  indicate  that,  on  average,  the 
respondents  fell  between  neither  disagree  or  agree.  There  are,  however,  many 
respondents  that  fear  a  protest,  and  this  can  have  an  effect  on  acquisition  strategies. 
Notably,  there  is  a  small  but  distinctly  separate  contingent  of  respondents  (16,  or  4.5 
percent)  who  did  not  fear  a  protest  at  all.  Consistent  with  the  qualitative  comments 
received  in  the  comments  field  of  the  survey  and  in  qualitative  interviews,  these 
strongly-held  perspectives  may  be  personal  philosophies  that  protests  are  a  natural 
part  of  the  acquisition  process.  In  other  words,  why  fear  what  is  expected? 
Importantly,  this  small  contingent  slightly  reduced  the  average  fear,  meaning  that  the 
average  of  the  remainder  of  the  sample  would  be  slightly  higher  at  20.41 . 
Nonetheless,  there  appears  not  to  be  a  strong,  pervasive  problem  of  a  fear  of  protest 
within  the  Navy. 
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Figure  5.  FEAR  Construct  Histogram 


A  low  level  of  a  fear  of  protest  could  be  explained  by  a  contracting  officer’s 
confidence  resulting  from  their  experience  or  in  their  ability  to  properly  conduct  a 
source  selection.  As  one  of  the  interview  participants  pointed  out, 

I  am  not  fearful  of  a  protest  because  if  you  protest  me  and  I  did  it  right,  what — 
the  protest  is  not  going  to  be  sustained,  it  is  going  to  be  denied.  So  the  fear  of  a 
protest  is  usually  not  present  other  than  the  delay  in  the  procurement  action  in  case 
it  is  an  urgent  action  or  something  that  needs  to  get  in  place  and  time.  It  may  delay 
the  action,  there  may  be  a  stay  of  some  sort,  but  other  than  that  avoiding  a  protest — 
just  doing  the  right  thing  avoids  a  protest. 

Conventional  wisdom  suggests  that  high  levels  of  exposure  to  something  can 
lower  the  amount  of  fear  one  has  in  it  occurring.  Out  of  an  average  of  13.12  years  of 
contracting  experience,  contracting  officers  only  experienced  an  average  of  1 .96  bid 
protests. 

2.  Insufficient  PALT  Planned 

Hypothesis  1:  Insufficient  PALT  has  a  direct  positive  relationship  on  the 
level  of  fear  of  protest. 

There  is  strong  evidence  that  suggests  insufficient  PALT  is  a  contributor  to 
the  level  of  fear  of  protest.  Although  the  relationship  is  significant,  the  coefficient  is 
low.  The  more  insufficient  the  planned  procurement  lead-time  is  thought  to  be,  the 
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level  of  fear  of  a  protest  increases.  Alternatively,  when  insufficient  PALT  decreases, 
the  level  of  fear  decreases. 

The  relationship  is  very  plausible.  When  acquisition  personnel  have  less 
planned  PALT  than  they  believe  is  necessary  to  properly  conduct  the  source 
selection,  there  is  less  time  to  adequately  follow  an  acquisition  strategy.  This 
insufficient  time  increases  the  concern  of  a  protest  because  internal  customers  and 
acquisition  personnel  may  unnecessarily  rush  the  acquisition  process.  This 
relationship  is  particularly  plausible  when  the  EOFY  is  approaching.  Customers  must 
obligate  their  remaining  funds  before  the  EOFY,  September  30,  or  lose  the 
remaining  funds.  In  some  instances,  customers  submit  to  the  contracting  agencies 
their  requirements  a  few  days  before  the  EOFY.  There  is  little  time  to  adequately 
complete  the  acquisition  process. 

3.  Contracting  Officer’s  Competence 

Hypothesis  2:  The  greater  a  contracting  officer’s  competence  level,  the 
lower  the  level  of  fear  of  protest. 

In  the  examination  of  a  contracting  officer’s  years  of  experience  as  an 
indicator  of  his  or  her  competence,  there  was  no  relationship  with  a  fear  of  protest. 
There  is  strong  evidence,  however,  that  a  contracting  officer’s  competence  in  terms 
of  the  number  of  source  selections  experienced  lowers  the  level  of  fear  of  bid 
protest.  The  coefficient  path  of  -.267  suggests  that  there  is  a  negative  relationship 
between  the  contracting  officer’s  source  selection  participation  and  the  level  of  a  fear 
of  protest. 

It  was  hypothesized  that  there  would  be  a  relationship  with  both  the  years  of 
experience  and  the  number  of  source  selections  experienced.  However,  the  lack  of  a 
relationship  with  regard  to  years  of  experience  can  be  explained  because  of  a  few 
likely  scenarios.  Experience  in  a  FAR  Part  15  source  selection  can  vary  greatly 
regardless  of  one’s  years  of  experience.  Some  may  have  been  in  the  contracting 
career  field  for  many  years  but  have  rarely  dealt  with  a  FAR  Part  15  source  selection 
because  they  have  worked  mainly  post-award  contract  administration,  pre-award 
simplified  acquisitions,  or  are  in  a  mission  area  whose  requirements  are  mostly  sole 
source  procurements  such  as  research  and  development.  The  number  of  source 
selections  is  a  better  determinant  for  measuring  competence  than  is  years  of  general 
contracting  experience.  The  research  supports  that  the  greater  the  quantity  of  FAR 
Part  15  source  selections  a  contracting  officer  has  experienced,  the  more 
comfortable  they  become  in  their  ability  to  work  through  the  proper  process  and 
ultimately  have  less  fear  of  a  bid  protest. 

The  empirical  data  also  did  not  support  the  DAWIA  certification  level  as  a 
surrogate  measure  of  competence.  No  relationship  between  DAWIA  certification 
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level  and  fear  of  protest  was  found.  There  can  very  well  be  a  contracting  officer  with 
a  level  two  DAWIA  certification  that  has  more  source  selection  experience  than  a 
level  three  DAWIA  certified  contracting  officer.  DAWIA  level  is  not  a  reliable  indicator 
of  experience,  capability,  and  therefore,  a  KO’s  confidence  in  ability. 

It  is  important  for  practitioners  to  understand  that  competence  comes  in  many 
forms  and  the  person  selected  for  a  source  selection  of  importance  should  not  be 
chosen  on  DIAWA  level  or  years  of  general  contracting  experience  alone;  but  they 
should  also  look  at  how  many  source  selections  the  contracting  officer  has 
experienced. 

4.  Requirement  Criticality/Importance 

Hypothesis  3:  The  level  of  requirement  criticality/importance  has  a 
positive  impact  on  the  level  of  fear  of  protest. 

There  is  overwhelming  evidence  that  requirement  criticality/importance  is  a 
contributor  to  the  level  of  fear.  The  relationship  for  Hypothesis  3  is  statistically 
significant  and  the  path  coefficient  is  positive  and  fairly  strong,  Therefore,  as  the 
level  of  the  requirements  criticality/importance  increases,  the  level  of  fear  of  protest 
increases.  There  is  much  more  concern  and  pressure  to  ensure  the  customer  gets 
the  requested  requirement  in  a  timely  manner  when  the  requirement  is  critically 
important  to  their  mission.  Protests  take  time  and  delay  the  customer’s  receipt  of  his 
or  her  required  products  or  services.  If  the  requirement  is  of  great  importance  to  the 
customer,  the  pressure  from  this  importance  is  felt  by  the  contracting  officer.  This 
can  increase  the  contracting  officer’s  level  of  a  fear  of  protest.  Managers  should  be 
aware  of  this  and  ensure  that  proper  planned  PALT  is  built  into  the  acquisition 
strategy  to  counter  the  increase  of  a  fear  of  protest  and  to  ensure  enough  time  for 
proper  peer  reviews. 

5.  Protest  Risk 

Hypothesis  4:  As  protest  risk  increases,  the  level  of  fear  of  protest 
increases. 

There  is  strong  evidence  suggesting  that  protest  risk  is  a  contributor  to  the 
level  of  fear  of  protest.  The  relationship  for  Hypothesis  4  is  statistically  significant, 
and  the  beta  coefficient  is  positive.  As  the  level  of  undesirable  consequences 
increases,  the  level  of  fear  increases.  There  is  a  perception  that  a  bid  protest  reflects 
poorly  on  the  contracting  officer,  as  demonstrated  by  the  following  quote  by  one  of 
the  contracting  officers  interviewed:  ‘“oh,  you  got  a  protest.  What  did  you  do  wrong?’ 
kind  of  thing.  I  don’t  think  there  is  validity  to  that,  but  I  think  that  sometimes  it  is 
looked  at,  oh  so  you  got  a  protest  so  maybe  they  didn’t  do  something  right  in  there.” 
Another  contracting  officer  interviewed  expressed  that,  “Well,  first  off  is  that  it  reflects 
poorly  on  the  command.  It  gives  the  command  a  bad  reputation  in  terms  of  being 
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able  to  follow  the  source  selection  plan,  for  example,  professionalism  of  the  contract 
specialist  and  contracting  officers  and  the  source  selection  team  and  all  of  the 
people  that  contribute  to  the  award  decision.  Secondly,  protests  they  take  a  lot  of 
time.” 

On  the  other  hand,  in  some  commands  it  is  considered  a  career  milestone  to 
have  experience  with  a  protest  for  promotion  to  GS-13,  as  was  expressed  in  our 
interviews.  Another  common  theme  throughout  our  interviews,  and  as  indicated  in 
our  survey  responses,  is  the  increased  workload  a  protest  demands.  Everyone  is 
busy,  and  the  last  thing  needed  is  additional  work.  Furthermore,  new  requirements 
do  not  stop  coming  just  because  a  protest  occurred;  a  protest  can  significantly 
increase  an  already  overburdened  work  load. 

There  are  several  consequences  to  the  acquisition  strategy  and  contract 
outcomes  as  a  result  of  fear.  The  following  hypotheses  focus  on  the  results  relating 
to  these  consequences. 

6.  Technical  Evaluation  Effectiveness 

Hypothesis  5:  The  level  of  fear  of  protest  has  a  negative  effect  on  the 
technical  evaluations  effectiveness. 

As  a  result  of  fear,  there  is  statistical  evidence  that  supports  a  negative  effect 
on  the  effectiveness  of  technical  evaluations.  The  negative  relationship  between  fear 
of  protest  and  effectiveness  of  technical  evaluations  is  small,  however.  The 
relationship  suggests  that  as  a  person’s  level  of  a  fear  of  protest  increased,  the 
effectiveness  of  the  technical  evaluation  decreased.  From  the  research  data,  on  a 
scale  of  one  to  seven  with  one  being  strongly  disagree  and  seven  being  strongly 
agree,  responses  averaged  4.5  on  this  scale  that  measured  “At  least  one  technical 
evaluator  expressed  concern  about  not  being  able  to  say  what  needs  to  be  said  in 
the  technical  evaluation.”  contracting  officers  somewhat  agreed  with  an  average  of 
4.63  on  the  scale  that  “At  least  one  technical  evaluator  was  concerned  that  the 
constraints  imposed  on  their  evaluations  impeded  his/her  ability  to  write  a 
meaningful  evaluation.”  This  suggests  that  the  technical  evaluators  do  not  believe 
the  process  is  sufficiently  optimized.  Additionally,  on  the  scale  for  “Upon  evaluation 
of  proposals,  at  least  one  technical  evaluator  expressed  a  need  to  change  at  least 
one  evaluation  criterion  or  its  definition;”  the  contracting  officer  somewhat  agreed 
with  this  with  an  average  of  4.53.  It  is  plausible  that  a  contracting  officer’s  fear  of 
protest  could  lead  them  to  change  the  wording  or  the  standards  of  the  technical 
evaluations.  To  a  certain  degree,  without  the  threat  of  a  bid  protest,  the  quality  of  the 
technical  evaluation  would  change.  The  statistical  analysis  suggests  that  the 
effectiveness  of  the  technical  evaluations  decreases  as  the  level  of  fear  increases. 
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7.  Source  Selection  Method  Fit/Appropriateness 

Hypothesis  6:  There  is  a  direct  positive  relationship  between  fear  of 
protest  and  the  Source  Selection  method  fit/appropriateness. 

Hypothesis  6  was  partially  supported  and  tests  were  consistent  with  the 
relationship  hypothesized.  The  hypothesis  measured  fear  of  protests  effect  on  both 
perceived  Source  Selection  Method  Appropriateness  and  on  multiple  aspects  of  the 
source  selection  strategy.  The  relationship  between  fear  of  protests  and  source 
selection  method  appropriateness  was  not  supported.  The  appropriateness 
construct  consisted  of  questions  relating  to  the  acquisition  strategy  used  on  the 
particular  source  selection.  Acquisition  strategy  is  a  broad  topic  that  encompasses 
decisions  made  by  various  stakeholders  early  on  in  the  process  of  fulfilling  a 
requirement.  It  is  plausible  that  these  decisions  are  made  so  early  that  the  factors 
that  contribute  to  fear  of  protest  have  not  been  solidified. 

Several  relationships  between  fear  of  protest  and  various  aspects  of  the 
source  selection  strategy  were  found,  however  perhaps  the  most  telling  evidence  of 
the  deleterious  effects  that  fear  of  protest  was  its  impact  on  LPTA  inappropriateness. 
Significant  findings  included 

•  Method  inappropriateness — While  it  represents  only  7.5  percent  of  the 
respondents  that  used  either  LPTA  or  trade-off  methods,  it  must  be 
noted  that  23  respondents  used  a  source  selection  method  that  they 
felt  was  to  some  degree  inappropriate.  The  relationship  between  fear 
of  protests  and  the  inappropriate  use  of  LPTA  was  significant.  The 
results  show  that  practitioners  with  high  fear  of  protests  are  four  times 
more  likely  to  inappropriately  use  LPTA. 

•  Negative  effects  on  satisfaction  with  discussions — significant 
relationships  were  found  between  fear  of  protests  and  satisfaction  with 
discussions.  When  fear  of  protests  exists,  the  level  of  satisfaction  to 
freely  and  openly  conduct  discussions  with  an  offeror  diminishes. 

While  the  effects  were  marginal,  the  mere  existence  of  this  relationship 
is  significant.  When  respondents  were  measured  for  the  extent  that 
they  would  change  discussions  if  there  were  no  ability  to  protest,  the 
results  were  also  significant.  The  results  indicated  there  would  be  a 
distinct  change  in  discussions  if  the  protest  system  did  not  exist. 

The  negative  effects  that  the  fear  of  protest  has  on  source  selection  decision¬ 
making  can  be  seen  most  in  the  variables  measured  to  test  this  hypothesis. 
Contracting  professionals  are  trained  and  trusted  to  make  decisions  that  support  the 
best  interest  of  the  government.  It  was  statistically  shown  that  fear  of  protests  does 
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have  a  negative  effect  on  acquisition  strategy  decisions.  However,  it  must  be  noted 
that  the  magnitudes  of  effects  is  not  substantial. 

8.  Contractor  Performance 

Hypothesis  7;  There  is  a  positive  relationship  between  the 
fit/appropriateness  of  a  source  selection  and  contractor  performance. 

Hypothesis  8:  There  is  a  positive  relationship  between  technical 
evaluation  effectiveness  and  contractor  performance. 

Although  source  selection  method  appropriateness  was  not  statistically 
significant  in  its  relationship  with  fear  of  a  protest,  Hypothesis  7  was  significant. 
There  is  a  positive  relationship  between  perceived  source  selection  method 
appropriateness  and  contractor  performance.  As  acquisition  strategies  are  properly 
tailored  to  fit  requirements,  contractor  performance  tends  to  increase  as  a  result.  In 
assessing  source  selection  method  appropriateness,  survey  respondents  were 
prompted  to  consider  the  buying  situation  (e.g.,  complexity,  dollar  value,  acquisition 
objectives,  contract  length,  performance  risk,  criticality  to  the  mission,  availability  of 
supply,  time  available  to  award  a  contract,  etc.).  They  were  also  asked  to  rate  how 
well  the  strategy  would  achieve  the  acquisition  objectives  and  how  well  the  strategy 
would  help  select  the  best  offeror. 

Hypothesis  8  was  also  statistically  significant.  As  the  effectiveness  of 
technical  evaluations  increases,  so  does  the  contractor  performance.  Hence,  as 
more  effective  technical  evaluations  are  generated,  the  more  apt  the  source 
selection  team  is  to  select  the  contractor(s)  that  can  perform  the  work  well.  The 
contractor  that  has  the  best  value  to  the  government  is  selected  for  the  requirement, 
which  will  ultimately  lead  to  greater  contractor  performance. 

9.  Buyer  Satisfaction 

Hypothesis  9:  There  is  a  direct  positive  relationship  between 
contractor’s  performance  based  on  the  source  selection  method  used 
and  the  Buyer’s  satisfaction. 

There  is  overwhelming  evidence  that  suggest  that  the  contractor’s 
performance  contributes  to  an  increase  in  the  buyer’s  satisfaction.  The  relationship 
suggests  that,  as  a  contractor’s  performance  increases,  buyer’s  satisfaction 
increases.  The  contracting  officer’s  responsiveness  and  willingness  to  address  the 
concerns  of  the  customer  can  lead  to  a  better  suited  acquisition  strategy  and  have  a 
positive  effect  on  customer  satisfaction.  The  contracting  officer  is  working  on  behalf 
of  the  customer  and  the  customer  is  the  lifeblood  of  the  contracting  activity.  It  is  in 
the  best  interest  of  the  contracting  activity  to  ensure  proper  contract  performance  by 
using  the  appropriate  contract  vehicle,  source  selection  method,  and  overall 
acquisition  strategy.  This  will,  in  turn,  improve  customer  satisfaction  and  increase 
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their  public  value  to  the  taxpayer.  Therefore,  it  is  imperative  for  contracting  officers  to 
select  the  appropriate  source  selection  method  to  choose  the  best  Contractor  to 
satisfy  the  buyer’s  requirements  and  expectations. 

10.  PALT  Actual 

Hypothesis  10:  There  is  a  direct  positive  relationship  between  fear  of 
protest  and  the  actual  PALT. 

There  is  statistical  evidence  that  suggests  that  a  fear  of  protest  contributes  to 
an  increase  in  actual  PALT.  The  data  suggests  that  there  is  a  slightly  greater  than  50 
percent  chance  that  the  actual  PALT  will  increase  as  a  result  of  fear  of  protest.  If 
contracting  officers  are  concerned  about  a  protest,  then  they  will  spend  more  time  to 
justify  their  decisions.  They  could  seek  assistance  from  other  acquisition  personnel 
for  their  opinion  and  recommendations,  which  can  increase  actual  PALT.  They  can 
also  refer  to  their  legal  advisors  and  supervisors  who  likely  recommend  additional 
steps  or  reworked  documents  that,  in  turn,  ultimately  increasing  actual  PALT.  From 
the  data,  the  average  PALT  planned  was  182.97  days  (6.09  months).  The  average 
actual  PALT  was  236.72  days  (7.89  months).  The  difference  is  53.75  days  (1.79 
months).  Added  time  is  added  money  (i.e.,  transaction  costs).  Thus,  efficiency  is 
compromised  with  greater  fear  of  protest.  The  data  suggests  that  some  of  this 
additional  time  can  be  attributed  to  a  fear  of  protest.  Although  not  tested,  this  can 
have  an  effect  of  customer  satisfaction  and  delay  mission  capabilities. 

1 1 .  Resources 

Hypothesis  11:  There  is  a  direct  positive  relationship  between  fear  of 
protest  and  the  number  of  Human  Resources  assigned  to  the  source 
selection  team. 

There  is  statistical  evidence  that  supports  a  direct  positive  relationship 
between  protest  fear  and  the  number  of  personnel  assigned  to  the  source  selection 
team  (boards,  contracting  team,  price  analysis  team,  past  performance  team,  legal 
team,  and  technical  evaluation  team).  Additionally,  the  research  data  captures  an 
average  of  8.96  people  are  on  the  source  selection  team.  Without  the  risk  of  protest, 
contracting  officers  believe  that  an  average  of  7.65  people  should  have  been  on  the 
source  selection  team.  The  difference  is  1 .33  people.  Additionally,  1 10  survey 
respondents  indicated  that  they  had  more  people  than  what  they  believed  was 
needed.  More  people  translate  into  more  transaction  costs.  The  results  suggest  that 
at  least  one  person  too  many  is  assigned.  Perhaps  more  attention  is  needed  to  not 
over-staff  teams  thereby  avoid  wasted  resources.  Additionally,  if  fear  of  protest  can 
be  reduced,  team  size  can  be  reduced  and  transaction  costs  can  be  avoided.  While 
these  salary  costs  may  be  dismissed  as  sunk  costs,  certainly  excess  personnel 
could  accomplish  other  pertinent  work  if  not  serving  on  the  source  selection  team. 
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12.  Transaction  Costs 

Hypothesis  12:  There  is  a  direct  positive  relationship  between  fear  of 
protest  and  transaction  costs. 

There  is  strong  evidence  that  a  fear  of  protest  causes  a  small  increase  in 
transaction  cost.  The  average  cost  per  transaction  was  $235,236  with  a  standard 
deviation  of  $291 ,620.  The  research  also  uncovered  that  for  every  dollar  awarded  in 
a  source  selection  there  is  an  average  of  0.08  cents  in  personnel  cost.  Without  a 
known  benchmark,  it  is  difficult  to  generalize  the  magnitude  of  these  costs. 

As  the  level  of  fear  of  a  protest  increase,  the  cost  devoted  to  the  source 
selection  process  increases  as  well.  Furthermore,  as  the  fear  of  a  protest  increases, 
so  do  the  iterations  of  common  source  selection  documents.  The  data  showed  that  a 
fear  of  protest  directly  caused  a  small  increase  in  the  amount  of  iterations  of  the 
following  documents:  source  selection  plan,  debriefing  scripts,  color  and  rating 
charts,  and  evaluation  debriefing  charts.  These  documents  must  be  completed  and 
reviewed  by  human  capital  costing  additional  time  and  money  devoted  to  a  source 
selection.  We  found  very  little  qualitative  evidence  that  transaction  costs  were  of  any 
concern  to  interview  informants.  The  maxim  appears  to  be  to  defend  the  acquisition 
against  a  protest  no  matter  the  cost  in  terms  of  time  and  effort. 

13.  Number  of  Awarded  Contracts 

Hypothesis  13:  The  higher  the  level  of  fear  of  protest,  the  greater  the 
number  of  awarded  contracts. 

There  is  no  statistical  evidence  to  suggest  that  the  higher  the  level  of  fear,  the 
greater  the  number  of  awarded  contracts.  From  the  data,  contracting  officers  are 
awarding  about  the  same  amount  of  contracts  that  they  planned.  An  average  of  2.24 
contracts  was  planned.  An  average  of  2.39  contracts  was  awarded.  There  is  a  small 
difference  of  0.15,  but  our  data  suggests  that  fear  is  not  a  factor  in  the  number  of 
awarded  contracts.  As  reported  in  the  trade  literature,  there  are  instances  of 
awarding  more  contracts  in  order  to  thwart  a  specific  protest,  but  this  practice 
appears  not  to  be  pervasive. 

14.  Contracting  Officer  Authority 

Hypothesis  14:  There  is  a  direct  negative  relationship  between  fear  of 
protest  and  the  contracting  officer’s  authority. 

Although  the  statistical  evidence  is  marginally  significant,  there  is  some 
evidence  that  fear  of  protest  diminishes  contracting  officers’  authority  (i.e. ,  discretion 
in  making  decisions). 

This  relationship  has  plausibility.  As  contracting  officers  are  more  concerned 
about  a  protest,  they  seek  assistance  in  avoiding  a  protest,  which  could  diminish 
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their  discretion  in  making  decisions.  If  a  contracting  officer  goes  to  the  legal  advisor, 
Legal  could  influence  the  contracting  officer’s  decision  through  their  legal  opinion 
and  recommendations.  Often,  Legal  and  committee  advisors  will  be  conservative 
and  recommend  wording  changes  to  documents,  changes  to  ratings,  amendments 
to  the  RFP  and  subsequent  invitations  for  revised  proposals,  and/or  further 
discussions  to  clear  up  any  uncertainty  in  evaluations. 

Additionally,  if  a  contracting  officer  has  a  high  level  of  fear  of  protest,  other 
acquisition  team  members  likely  do  as  well.  A  high  concern  of  a  protest  could  cause 
others  not  to  empower,  trust,  or  support  the  contracting  officer  with  making  the 
required  decisions,  which  could  diminish  a  Contracting  officer’s  discretion  in  making 
decisions.  During  the  interviews,  it  appeared  that  the  level  of  which  the  contracting 
officer  felt  compelled  to  take  the  advice  of  legal  as  “gospel”  depended  on  the 
contracting  officer.  Some  felt  as  though  they  were  compelled  to  take  the  advice  of 
legal  and  some  felt  it  was  within  their  discretion  to  do  what  they,  as  the  contracting 
officer,  thought  was  the  best  course  of  action. 

It  is  ultimately  obviously  the  contracting  officer’s  decision  and  it  basically 
depends  on  what  their  comfort  level  is  in  terms  of  some  won’t  turn  their  head  without 
getting  an  opinion  from  legal  and  making  sure  that  they  sign  off  on  it  and  other  ones 
basically  believe — and  I  agree — that  legal  is  an  advisory  role,  obviously. 

It  has  been  noted  during  our  interviews  that  the  legal  department  is  the  most 
concerned  about  a  protest.  This  makes  sense  since  they  will  carry  a  lot  of  the  work 
involved  in  a  protest  if  received.  It  has  also  been  stated  in  our  interviews  that  legal 
advocates  for  a  LPTA  source  selection  methodology.  Since  our  data  suggests  that 
there  is  a  very  weak  negative  relationship  with  a  fear  of  protest  and  contracting 
officer’s  authority,  we  can  infer  that  the  contracting  officers  are  not  unnecessarily 
influenced  by  legal,  but  there  were  13  cases  in  our  350  respondents  in  which  the 
contracting  officer  chose  LPTA  even  though  they  believed  it  was  not  the  best  course 
of  action. 

C.  ANSWERS  TO  RESEARCH  QUESTIONS 

1.  Research  Question  1:  Are  contracting  officers  sufficiently 

concerned  about  bid  protests  to  alter  acquisition  strategies? 

The  answer  to  this  research  question  is  ‘yes.’  U.S.  Navy  contracting  officers 
change  aspects  of  acquisition  strategies  to  avoid  a  protest.  However,  U.S.  Navy 
contracting  officers  appear  not  to  be  overly  concerned  about  a  bid  protests.  The 
concerns  of  protest  exist,  but  overall  the  magnitude  is  not  excessive.  There  is  little 
evidence  from  the  research  suggesting  that  contracting  officers  drastically  alter 
acquisition  strategies  to  avoid  a  protest. 
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There  is  evidence  to  suggest  that  a  fear  of  protest  can  alter  an  acquisition 
strategy.  The  350  survey  respondents,  who  have  an  average  of  13.12  years  of 
contracting  experience,  were  asked  throughout  their  career,  how  many  times  they 
have  awarded  a  task/delivery  order  against  an  IDIQ  contract  (or  Blanket  Purchase 
Agreement  [BPA])  in  order  to  avoid  a  bid  protest.  The  data  shows  that  88 
respondents  had  done  so  throughout  their  career  with  4,139  contracting  actions.  It 
was  also  shown  that  67  respondents  avoided  a  protest  using  a  sole  source  award 
throughout  their  career  (101 1  contracting  actions)  and  80  respondents  had  modified 
an  existing  contract  throughout  their  career  in  order  to  avoid  a  protestable 
competitive  procurement  (1,065  contracting  action).  With  350  respondents  that  have 
an  average  of  13.12  years  of  experience,  this  equals  4,784.5  years  of  total 
experience.  The  total  contracting  actions  from  the  three  previously  discussed  survey 
questions  totals  6,215.  This  is  just  under  1.3  redirected  contracting  actions  per  year 
of  experience.  There  are  consequences  of  a  fear  of  protest  that  our  research 
revealed.  These  consequences  are  discussed  in  research  question  three. 

2.  Research  Question  2:  If  so,  what  factors  affect  protest  “fear”? 

Based  on  the  regression  model,  insufficient  PALT,  protest  risk,  the  number  of 
source  selections  a  contracting  officer  previously  experienced,  and  the  requirement 
criticality/importance  are  the  factors  that  affect  a  fear  of  protest.  Neither  a 
Contracting  officer’s  years  of  contracting  experience  nor  their  DAW  I A  certification 
level  affected  the  fear  of  protest. 

3.  Research  Question  3:  What  are  the  consequences  to  the 
acquisition  strategy  and  contract  outcomes? 

Based  on  the  significant  levels,  a  fear  of  protest  can  have  negative 
consequences  on  the  effectiveness  of  technical  evaluations  leading  to  a  decrease  in 
contractor  performance  and,  in  turn,  buyer  satisfaction.  An  increase  of  fear  of  protest 
has  a  negative  effect  on  technical  evaluations  effectiveness.  Technical  evaluation 
effectiveness  has  a  positive  influence  on  contractor  performance  which,  in  turn,  is 
positively  related  to  buyer  satisfaction.  Technical  evaluation  effectiveness 
considered  elements  such  as  (a)  being  able  to  say  what  needs  to  be  said,  (b)  being 
able  to  write  a  meaningful  evaluation,  and  (c)  a  need  to  change  an  evaluation  factor 
or  its  definition. 

The  perceived  appropriateness  of  the  source  selection  method  will  also  have 
a  positive  impact  on  contractor  performance.  The  more  appropriate  the  source 
selection  method  used,  the  higher  the  contractor  performance  leading  to  greater 
buyer  satisfaction.  A  notable  finding  from  this  research  is  that  the  inappropriate  use 
of  LPTA  was  shown  to  have  a  substantial  negative  impact  on  contractor 
performance. 
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Actual  PALT  can  increase  as  a  result  of  a  fear  of  protest.  The  research  data 
revealed  that  there  is  a  50  percent  chance  that  PALT  will  increase  as  a  result  of  fear. 
Thus,  efficiency  is  compromised  with  greater  fear  of  protest.  The  data  suggests  that 
some  of  this  additional  time  can  be  attributed  to  a  fear  of  protest.  Although  not 
tested,  this  can  have  an  effect  of  customer  satisfaction  and  delay  mission 
capabilities. 

Contracting  officers  perceive  their  authority  to  decrease  as  their  fear  of 
protest  increases.  A  plausible  explanation  can  be  that  as  contracting  officers  are 
more  concerned  about  a  protest;  they  seek  assistance  in  avoiding  a  protest,  which 
could  diminish  their  discretion  in  making  decisions.  Additionally,  a  high  concern  of  a 
protest  could  cause  others  not  to  empower,  trust,  or  support  the  contracting  officer 
with  making  the  required  decisions,  which  could  diminish  a  contracting  officer’s 
discretion  in  making  decisions. 

Fear  of  protest  also  has  a  negative  impact  on  the  level  of  satisfaction  with 
discussions.  As  fear  increases,  the  contracting  officer’s  satisfaction  with  discussions 
decreases.  This  is  an  indicator  of  suboptimal  discussion  that  could  have  implications 
for  the  awarded  contract  and,  more  importantly,  contractor  performance.  Another 
consequence  of  a  fear  of  protest  is  the  transaction  cost  and  time  associated  with  the 
source  selection.  As  a  fear  of  protest  increases,  so  do  the  iterations  of  some 
documents,  such  as  source  selection  plan,  debriefing  scripts,  color  and  rating  charts, 
and  evaluation  debriefing  charts.  This  can  lead  to  additional  time  and  money  (i.e., 
opportunity  costs)  spent  on  the  requirement. 

D.  IMPLICATIONS  AND  RECOMMENDATIONS  FOR  PRACTITIONERS 

This  section  discusses  some  of  the  possible  implications  this  study  revealed 
and  what  managers  and  practitioners  can  learn  from  them.  As  discussed  earlier,  the 
magnitude  of  fear  of  protest  does  not  appear  to  be  a  pervasive  problem  within  the 
Navy’s  contracting  community,  but  there  are  many  contracting  officers  that  allow  this 
fear  to  affect  elements  of  their  acquisition  strategy.  There  were  23  survey 
respondents  (7.49  percent)  that  revealed  that  the  source  selection  method  used  was 
to  some  degree  inappropriate.  While  this  proportion  appears  insignificant,  it  can  be 
argued  that  any  single  instance  of  an  inappropriate  source  selection  method  gives 
room  for  pause  to  see  what  is  driving  this  inappropriateness.  As  mentioned  by  an 
interview  participant,  “I  will  tell  you,  legal  pushes  the  LPTA.  They  push  it  a  lot.” 
Another  interviewee  mentioned,  “at  this  juncture,  there  are  too  many  hands  in  the 
soup,  and  the  Procuring  Contracting  Officer  (PCO)  authority  has  been  diminished. 
Attorneys  need  to  resume  the  role  of  counselors  again.”  Fear  of  protests  does  not 
generate  good  contracts,  or  in  this  case  can  prevent  any  award  from  being  made. 
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Recommendation  1:  Agencies  should  foster  a  culture  that  empowers 
the  contracting  officer  to  perform  the  job  they  are  trained  to 
accomplish. 

Since  the  source  selection  method  is  not  a  matter  of  legal  sufficiency,  Legal 
should  not  determine  the  source  selection  method  used,  nor  should  they  encourage 
any  particular  source  selection  method  such  as  LPTA.  Selecting  the  source  selection 
method  is  a  contracting  officer’s  decision  based  on  experience,  knowledge,  and 
professional  judgment;  contracting  officers  should  feel  empowered  and  should  in  no 
way  be  influenced  against  their  better  judgment.  Otherwise,  Federal  Government 
agencies  may  have  a  professional  field  with  a  high  degree  of  accountability  but 
subtly-diminished  authority.  Acquisition  leaders  should  reassure  their  contracting 
officers  that  they  have  the  ultimate  say  in  the  source  selection  method  employed, 
and  that  legal’s  role  is  to  advise  and  to  clarify  the  many  legal  issues  that  contracting 
within  the  government  entails.  If  the  contracting  officer  does  not  have  the  authority  or 
believes  that  the  legal  department  has  the  ultimate  say,  it  would  be  less  expensive  to 
the  government  to  hire  clerks  or  paralegals  to  accomplish  the  contracting  mission. 
The  judgment  of  the  contracting  officer,  the  person  actually  signing  the  contract, 
should  be  respected  by  management  and  management  should  encourage  a  culture 
that  empowers  the  contracting  officer. 

Undue  influence  from  management  can  also  lead  to  an  inappropriate  source 
selection  method.  There  is  pressure  applied  to  management  to  reduce  their  unit’s 
average  time  for  procurements.  This  can  lead  to  pressure  from  management  to  the 
contracting  officer  resulting  in  the  overuse  of  LPTA.  Management  should  resist  this 
pressure.  Our  data  suggests  that  LPTA  inappropriateness  can  have  a  negative 
effect  on  contractor  performance  that  leads  to  less  buyer  satisfaction.  Ultimately  the 
best  course  of  action  is  to  use  the  source  selection  method  that  best  suits  the 
requirement. 

Recommendation  2:  Agencies  should  establish  a  standard  policy  for 
PALT based  on  dollar  value  and  source  selection  method. 

Planned  PALT  is  often  a  result  of  the  internal  customer’s  situation.  It  may  be  a 
requirement  that  is  needed  within  three  weeks  or  three  months,  but  most  everyone 
wants  their  requirement  fulfilled  as  quickly  as  possible.  It  is  important  that  the 
contracting  officer  has  the  ability  to  manage  the  expectations  of  the  customer  and 
that  the  contracting  officer’s  management  supports  them  on  this  issue.  If  planned 
PALT  is  less  than  what  the  contracting  officer  believes  to  be  sufficient,  there  are 
implications  that  a  fear  of  a  protest  will  increase.  This,  in  turn,  can  ultimately  have  a 
negative  effect  on  technical  evaluation  effectiveness  and  ultimately  on  the 
contractor’s  performance.  A  possible  solution  to  this  is  to  have  an  instruction  that 
sets  PALT  by  dollar  value  and  by  source  selection  method.  A  higher  dollar 
requirement  that  is  best  procured  using  a  trade-off  approach  will  take  more  time  and 
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should  be  built  in  to  the  planned  PALT.  The  contracting  officer  should  have  the 
flexibility  to  deviate  from  these  predetermined  PALT’s  based  on  their  experience  and 
judgment. 

Recommendation  3:  Agencies  should  establish  a  training  program  to 

supplement  FAR  Part  15  source  selection  experience. 

The  competence  level  of  the  contracting  officer  is  validly  measured  in  terms  of 
the  number  of  source  selections  experienced  (and  not  raw  years  of  contracting 
experience  or  DAWIA  certification  level).  As  the  number  of  source  selections 
conducted  increases,  the  level  of  fear  of  protest  decreases.  Therefore,  efforts  should 
be  made  to  increase  the  number  of  source  selections  experienced  by  contracting 
officers.  Of  the  350  survey  respondents,  it  was  found  that  the  average  years  of 
experience  are  13.12  with  an  average  of  18.31  source  selection  participations.  That 
is  just  under  1 .40  source  selections  a  year.  There  is  no  equal  alternative  to  on-the- 
job-training  (OJT),  but  source  selection  simulations  and  scenario-based  training 
could  be  utilized  as  an  alternative  and  as  a  supplement  to  OJT.  If  the  acquisition 
community  is  relying  solely  on  OJT,  it  can  take  a  contracting  officer  far  too  long  to 
gain  an  adequate  level  of  competence  with  FAR  Part  15  source  selections.  In 
addition,  when  selecting  a  contracting  officer  for  a  source  selection,  it  is  important  to 
realize  that  years  of  contracting  experience  and  DAWIA  certification  level  do  not 
represent  his  or  her  level  of  competence. 

As  criticality  and  importance  of  the  requirement  increases,  so  does  the 
contracting  officer’s  level  of  a  fear  of  protest.  As  fear  increases,  buyer  satisfaction 
can  be  negatively  altered  through  the  inappropriate  use  of  source  selection  methods 
and  the  level  at  which  they  effectively  communicate  with  the  offerors.  Therefore,  the 
criticality  and  importance  of  the  requirement  should  be  taken  into  account  when 
assigning  a  contracting  officer  to  serve  as  the  PCO  for  the  source  selection.  Perhaps 
a  PCO  with  greater  source  selection  experience  (i.e.,  which  lowers  fear  of  protest) 
can  offset  the  higher  fear  of  protest  associated  with  more  critical  requirements. 

As  a  fear  of  protest  increases,  the  effectiveness  of  technical  evaluations 
decreases.  Recall  that  this  construct  encapsulated  situations  where  technical 
evaluators  expressed  concerns  that  (a)  they  did  not  have  the  latitude  to  say  what 
needed  to  be  said  in  their  evaluations,  (b)  constraints  impeded  the  ability  to  write  a 
meaningful  evaluation,  and  (c)  an  evaluation  factor  or  its  definition  needed  to  be 
changed  after  receipt  of  proposals.  This  should  be  cause  for  alarm  since  the 
purpose  of  a  source  selection  is  to  attain  a  needed  performance  level  outcome  at  a 
reasonable  price/cost  while  mitigating  risk.  Technical  evaluators  via  the  contracting 
officer  should  have  the  latitude  and  the  freedom  to  halt  a  source  selection  to  ensure 
the  customer  and  the  tax  payer  are  getting  the  best  contracted  outcome  (i.e.  value) 
for  the  invested  dollars;  though  this  recommendation  is  time  consuming  and  is  often 
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not  an  option.  Evaluators,  however,  can  reduce  the  risk  of  this  occurring  by 
becoming  part  of  the  crafting  of  Section  M  criteria.  Is  a  FAR  Part  15  full  trade-off 
source  selection  too  cumbersome  to  ensure  we  are  getting  the  best  requirement 
outcome?  Is  the  transaction  cost  of  full  trade  off  too  high?  It  may  be  that  the 
acquisition  rules  on  evaluations  and  the  protest  implications  of  them  are  too  strict 
and  FAR  part  15  may  need  another  reevaluation  to  allow  for  more  flexibility.  Our 
research  suggests  that  as  a  contracting  officer’s  fear  of  a  protest  increases,  there  is 
a  decrease  in  technical  evaluation  effectiveness  leading  to  a  decrease  in  contractor 
performance  and  less  buyer  satisfaction. 

Recommendation  4:  The  DAU  should  establish  a  prerequisite  online 

training  program  for  technical  evaluation  team  members. 

Additional  training  for  the  technical  evaluators  and  for  the  technical  evaluation 
team  is  required  to  increase  their  level  of  competence  within  the  evaluation  process. 
The  evaluation  process  has  many  people  that  are  not  necessarily  familiar  with  the 
rules  that  govern  this  process,  and  it  is  ultimately  the  responsibility  of  the  contracting 
officer  to  ensure  that  the  technical  team  is  aware  and  its  members  are  sufficiently 
trained  to  handle  the  evaluation  process.  The  contracting  officer  should  not  be  solely 
responsible  to  train  the  technical  evaluators,  and  all  people  involved  in  the  process 
should  have  a  minimum  training  requirement  that  educates  them  on  the  process.  All 
members  involved  in  the  technical  evaluation  process  should  have  a  minimum 
uniform  standard  of  training.  A  DOD  or  a  DAWIA  certification  could  give  the 
contracting  officer  the  confidence  that  the  technical  evaluation  team  has  the  proper 
level  of  knowledge  to  carry  out  the  evaluation  function.  If  the  contracting  officer  is 
comfortable  with  the  level  of  knowledge  then  he  or  she  can  focus  on  guiding  and 
advising  the  technical  evaluators  and  spend  less  time  elevating  their  baseline 
knowledge  of  the  process.  This  can  reduce  the  fear  a  contracting  officer  has  of  a 
protest  and  increase  the  effectiveness  of  technical  evaluations.  This  can,  in  turn, 
lead  to  increased  contractor  performance  and  buyer  satisfaction. 

E.  LIMITATIONS  OF  THE  STUDY 

This  study  had  several  limitations.  First,  this  study  was  limited  to  the  Navy’s 
acquisition  workforce.  Thus,  while  the  fear  of  a  protest  phenomenon  has  been 
discussed  in  the  context  of  all  Federal  contracting,  these  study  results  may  not  be 
generalized  beyond  the  U.S.  Navy.  Respondents  were  asked  to  answer  questions 
that  reflected  only  their  most  recent  completed  FAR  Part  15  source  selection  above 
$150,000  in  which  they  participated.  On  certain  questions,  respondents  provided 
estimated  responses.  For  example,  a  respondent  stated  that  a  GS-15  Supervisor 
spend  about  10  percent  of  one  year’s  time  dedicated  to  the  source  selection.  It  may 
be  difficult  for  respondents  to  estimate  accurate  times;  therefore,  these  estimated 
responses  may  not  be  precise.  However,  these  estimates  were  useful  in  establishing 
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a  benchmark  to  understand  the  monetary  cost  incurred  to  avoid  a  protest.  Activity- 
based  costing  using  a  database  (i.e.,  the  Navy’s  Standard  Labor  Data  Collection  and 
Distribution  Application  SLDCADA))  to  retrieve  data  that  captured  the  amount  of 
time  a  government  employee  worked  on  a  particular  source  selection  could  reveal  a 
more  accurate  monetary  cost  to  avoid  a  protest. 

Additionally,  contracting  officers  might  not  have  a  true  sense  of  buyer 
satisfaction.  Customers  (end-users)  may  not  share  their  satisfaction  or 
dissatisfaction  with  a  Contractor  to  the  contracting  officer  unless  there  are  significant 
problems.  If  the  problems  are  minor  and  can  be  worked  out  with  the  contracting 
officer  representatives  (COR)  or  customer,  the  contracting  officer  might  not  have 
knowledge  of  the  situation.  Secondly,  this  research  was  exploratory.  The  survey 
length  was  a  chief  concern,  which  we  suspect  resulted  in  a  low  response  rate.  Some 
of  the  acquisition  strategy  variables  relied  on  single  indicators  (e.g.,  dissatisfaction 
with  discussions,  appropriateness  of  source  selection  method  and  etc.)  Finally,  the 
responses  from  the  survey  were  subjected  to  the  problems  associated  with  self- 
reports  in  organizational  research  such  as  common  method  bias  (Podsakoff  & 

Organ,  1986). 

F.  AREAS  FOR  FURTHER  RESEARCH 

This  research  only  focused  on  a  few  factors  associated  with  fear  of  protest. 
According  to  the  models’  low  adjusted  R-square  values;  there  are  other,  unexplored 
factors  that  contribute  to  the  fear  of  protest.  Because  of  time  and  survey  constraints, 
this  research  was  not  able  to  examine  all  possible  factors.  Therefore,  it  is 
recommended  that  future  research  in  the  below  areas  be  done  to  enhance  our 
understanding  of  the  causes  and  effects  of  a  fear  of  protest. 

1.  Risk  Tolerance 

There  could  be  a  relationship  between  a  person’s  level  of  risk  tolerance  and 
the  level  of  fear.  If  a  person  is  risk-averse,  that  person  will  do  certain  actions  to 
reduce  the  risk.  In  terms  of  a  protest,  a  risk-averse  person  may  have  more  concern 
of  a  protest.  It  is  possible  that  the  risk  averse  person  would  change  the  uncertainty 
of  having  a  protest  to  the  certainty  of  not  having  a  protest.  If  a  person  has  a  high  risk 
tolerance,  then  that  person  is  willing  to  accept  risks.  That  person  might  not  be  as 
concerned  about  a  protest.  A  protest  to  that  person  “is  the  cost  of  doing  business.” 
Agencies  should  balance  a  person’s  level  of  risk  tolerance  against  the  procurement. 
Agencies  can  lower  transaction  costs  by  accepting  more  risk  and  by  assessing  the 
risk  tolerance  of  the  team  members  to  ensure  appropriate  risk  comfort  levels  are 
appropriate  for  the  requirement.  If  the  procurement  has  a  low  risk  of  protest,  a  high- 
risk  tolerance  person  should  handle  the  procurement.  The  same  is  true  for 
procurement  with  a  high  risk  of  protest;  a  risk-averse  person  should  handle  the 
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procurement.  The  risk-averse  person  could  take  the  extra  time  and  resources  to 
ensure  the  source  selection  is  sound. 

2.  Previous  Experience  with  a  Protest 

Fear  is  a  natural  reaction  to  an  unknown  threat.  From  the  data,  contracting 
officers  have  experienced  only  an  average  of  1 .96  protests  throughout  their  career. 
Many  contracting  officers  never  experience  a  protest.  Contracting  officers 
understand  the  general  repercussions  of  a  protest  through  press  accounts,  GAO 
reports,  and  interaction  with  those  who  have  experienced  one.  When  contracting 
officers  experience  a  protest,  their  view  of  the  protest  process  can  change.  It  is  a 
logical  conclusion  that  if  a  contracting  officer  has  experience  with  protest,  they  may 
be  less  fearful  of  a  protest.  However,  it  is  possible  that  having  experienced  a  protest, 
the  contracting  officer  might  be  more  inclined  to  alter  acquisition  strategies  to  avoid 
the  painful  experience  of  additional  work,  shame,  and  career  repercussions.  The 
implication  is  for  agencies  to  ensure  their  contracting  officers  have  personal 
experience  in  dealing  with  a  protest.  Requiring  contracting  officers  to  have  personal 
experience  could  be  a  difficult  task  because  the  rational  thought  is  to  prevent  a 
protest  from  happening.  If  an  agency  has  a  protest,  the  contracting  officers  with 
more  experience  of  a  bid  protest  should  actively  mentor  and  engage  less  experience 
contracting  officers  in  the  protest  adjudication  process.  Instead  of  having  solely  the 
experienced  contracting  officer,  or  the  assigned  procuring  contracting  officer,  other 
contracting  personnel  could  be  engaged  in  the  protest  as  well.  This  mentorship 
could  be  structured  as  an  apprenticeship.  Further,  given  the  way  the  internet  can 
close  the  geographic  divide,  apprentice  contracting  personnel  could  be  involved 
regardless  of  physical  separation  from  the  protested  contracting  activity.  The 
additional  contracting  personnel  on  the  protest  adds  additional  cost,  but  the  benefit 
of  gaining  firsthand  knowledge  and  experience  could  outweigh  the  costs  for  the 
agency.  Currently,  government  agencies — more  specifically,  contracting  activities 
within  them — generate  a  plethora  of  source  selection  experience.  However,  that 
experience  is  not  systematically  leveraged  to  more  quickly  build  more  vast 
competence  among  the  entire  contracting  workforce.  Is  there  a  way  to  learn  from 
others’  mistakes,  thereby  averting  repeats? 

3.  Inappropriate  Source  Selection  Method 

There  could  be  numerous  reasons  why  source  selection  teams  used  source 
selection  methods  that  were  perceived  to  be  inappropriate.  The  data  shows  that  13 
out  of  133  respondents  reported  that  their  use  of  LPTA  was  to  some  degree 
inappropriate.  Additionally,  10  respondents  out  of  174  reported  that  their  use  of  a  full 
trade-off  was  to  some  degree  inappropriate.  Additional  research  is  needed  to  narrow 
down  the  specific  causes  of  the  inappropriateness  of  the  source  selection  method.  It 
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could  have  been  solely  because  of  fear  of  protest,  time  constraints,  guidance  from 
senior  management  on  the  best  source  selection  to  use  for  that  particular 
procurement,  or  a  combination  thereof.  There  is  the  possibility  that  the  contracting 
officer’s  personal  perspective  could  have  played  a  role  in  determining  whether  the 
source  selection  used  was  appropriate  or  not.  If  there  are  systematic  circumstances 
on  why  contracting  officers  are  using  LPTA  or  trade-off  inappropriately,  agencies 
should  explore  and  implement  measures  to  reduce  the  circumstances  that  contribute 
to  inappropriate  use.  Possible  reasons  for  inappropriate  use  are: 

•  To  avoid  a  protest 

•  To  avoid  high  transaction  costs  associated  with  a  full  trade-off 

•  Lack  of  experience  with  full  trade-offs 

•  A  lack  of  understanding  of  the  true  complexity  of  the  requirement 

If  some  of  the  aforementioned  reasons  are  why  inappropriate  source 
selections  are  used,  this  could  be  fuel  for  further  reform  FAR  Part  15.  Perhaps  strict 
rules  are  dissuading  source  selection  teams  from  using  the  most  valuable  sourcing 
tool  available  to  them  (i.e. ,  full  trade-off).  Perhaps  the  complexity  of  discussions 
causes  source  selection  teams  to  forego  gaining  further  insight  into  offerors’ 
proposals. 

4.  Awarding  on  Initial  Proposals 

According  to  Mr.  Gordon,  “Contracting  officers  prefer  to  make  award  based 
on  initial  proposals,  rather  than  to  conduct  discussions,  because  they  fear  that 
discussions  with  offerors  are  a  legal  minefield,  such  that  conducting  discussions  will 
increase  the  likelihood  of  a  bid  protest  and  improve  the  protester’s  chances  of 
prevailing  if  a  protest  is  filed.”  (Gordon,  2013,  p.  37)  This  research  did  not  examine 
the  likelihood,  significance,  or  magnitude  of  the  relationship  between  fear  of  protest 
and  award  of  contracts  based  on  the  initial  proposal.  Awarding  contracts  based  on 
initial  proposals  could  have  secondary  and  tertiary  effects.  There  could  be  a 
significant  link  between  fear  of  protest  to  awarding  contracts  from  initial  proposals. 
That  link  could  extend  to  the  source  selection  method  fit,  and  then  to  the  Contractor 
Performance,  which  could  ultimately  have  an  effect  on  buyer’s  satisfaction.  If  there  is 
a  significant  relationship  of  fear  of  protest  and  the  award  of  initial  proposals,  then 
contracting  agencies  should  reduce  the  practice  because  discussions  are  beneficial 
to  the  contractor  and  the  agency,  which  would  be  beneficial  to  the  buyer. 

G.  SUMMARY 

In  conclusion,  fear  of  the  real  or  perceived  consequences  of  receiving  a  bid 
protest  exists.  U.S.  Navy  contracting  officers  have  some  concern  of  protests  and  this 
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can  be  linked  to  certain  consequences  on  acquisition  strategies.  However,  the  fear 
of  protest  appears  not  to  be  a  substantial  issue  in  the  Navy.  Across  five  survey 
questions,  the  average  score  was  a  4.08  (just  beyond  the  middle)  on  a  one  to  seven 
point  scale.  The  score  would  indicate  that,  on  average,  the  respondents  fell  between 
neither  “disagree  nor  agree”  and  “somewhat  agree.”  There  are,  however,  instances 
where  aspects  of  acquisition  strategies  are  altered  and  this  behavior  is  statistically 
linked  to  the  fear  of  protest.  Given  the  low  r-squared  values,  however,  factors  other 
than  a  fear  of  protest  are  also  culprits.  Further  research  is  needed  to  ascertain  these 
other  culprits,  then  compare  the  relative  effects  of  fear  of  protest  among  other 
factors.  Nonetheless,  there  is  enough  qualitative  and  quantitative  empirical  evidence 
to  suggest  that  fear  of  protest  can  impact  what  would  otherwise  be  prudent  business 
decisions.  The  greatest  concerns  are  a  few  instances  of  inappropriate  uses  of  LPTA 
and  the  reduced  technical  evaluation  effectiveness  attributed  to  fear  of  protests.  If 
fear  waters  down  the  source  selection,  hindering  its  ability  to  distinguish  between  the 
true  value  of  offers,  then  contracting  officers  must  ask  themselves  why  go  through 
the  trouble  of  a  best-value  source  selection?  Could  contracting  officers  simply  award 
to  the  low  bidder?  To  what  extent  is  the  set  of  stringent  source  selection  rules  driving 
the  acquisition  team  to  this  result  by  default  (i.e.,  regardless  of  source  selection 
method  actually  employed)?  Thus,  for  the  sake  of  stringent,  fairness-based  rules, 
contracted  outcomes  can  be  compromised.  Whether  the  tradeoff  is  prudent  remains 
to  be  determined.  Further  research  is  needed  to  ascertain  these  other  culprits,  then 
compare  the  relative  effects  of  fear  of  protest  among  other  factors. 
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APPENDIX  A. 


SURVEY  INSTRUMENT 


NPS  Enterprise  Survey  -  Bid  Protest 


5/12/14.  10:34  PM 


Bid  Protest 


There  are  115  questions  in  this  survey 


Consent 


[] 

By  clicking  on  the  "Yes"  button,  I  am  acknowledging  that  I  have  read  and 
understand  this  information,  that  I  understand  the  nature  and  purpose  of  this  study 
-  including  its  risks  and  benefits,  and  that  I  agree  to  voluntarily  participate  in  this 
online  survey.  I  also  understand  that  I  may  discontinue  at  any  time  simply  by 
exiting  this  website. 

This  survey  meets  the  requirements  of  OPNAV  5300. 8C  and  does  not  require  a 
formal  RCS  number  since  it  meets  the  criterion  of  a  task  analysis.  It  has  been 
reviewed  and  approved  by  the  Navy  Survey  Review  Program  Manager.  * 

Please  choose  only  one  of  the  following: 

D  Yes 

O  No 


http*://uirvey.nps.edu/adfnin/print&Me&urvey/u/index/surveyid/714497 
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NPS  Enterprise  Survey  -  Bid  Protest 


S/12/14,  10:34  PM 


Group  1 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  agree"  and  7  represents  "Strongly  disagree",  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  disagree"  and  7  represents 
"Strongly  agree",  rate  the  following  statements. 

At  some  point  during  the  development  of  the  acquisition  strategy  or  the  source 
selection  process,  I  worried  about  recieving  a  bid  protest.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


httpv//uirvey.np&.edu/admin/print&Mesurv«’y/&a/index/surveyid/714497  Page  2  of  54 
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NPS  Enterprise  Survey  -  Bid  Protest 


5/12/14,  10:34  PM 


[]I  was  concerned  that  the  contract  award  would  be  protested.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[]I  was  anxious  to  get  beyond  the  10-day  point  after  contract  award  (or 
debriefings)  to  determine  whether  or  not  the  contract  would  be  protested.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


httpi://Mirvey.nps.edu/Admin/prinub4esurv«ry/M/in<Sex/surveytd/714497 
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NPS  Enterprise  Survey  -  Bid  Protest 


S/12/14, 10:34  PM 


□  Receiving  a  bid  protest  would  have  been  among  the  worst  things  that  could 
happen.  * 

Please  choose  only  one  of  the  following: 

O  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


https://survey.nps.edu/admin/printAMesurvey/sA/imSex/suiveYid/714497 
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NPS  Enterprise  Survey  -  Bid  Protest 


S/12/14,  10:34  PM 


Group  2 


On  a  scale  of  1  to  1,  where  1  represents  “Not  at  all  concerned"  and  7  represents  "Extremely  concerned",  rate  the  following. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Not  at  all  concerned"  and  7  represents 
"Extremely  concerned",  rate  the  following. 


During  the  development  of  the  acquisition  strategy  and  throughout  proposal 
evaluation,  to  what  extent  were  you  concerned  that  an  offeror  might  protest  the 
contract  award?  * 

Please  choose  only  one  of  the  following: 

C)  1  Not  at  all  concerned 


Q  2 

Q  3 


Q  4  Concerned 


Q  5 
D  6 


D  7  Extremely  concerned 


https:  /  /  Mjrvey.npi.edu/Admin  /  print  aMeiurvey  /ia  /index/iurveyid/  7144  97 
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-IPS  Enterprise  Survey  -  Bid  Protest 


5/12/14, 10:34  PM 


□ 


During  the  development  of  the  acquisition  strategy  and  throughout  proposal 
evaluation,  to  what  extent  was  at  least  one  other  member  of  the  source  selection 
team  concerned  that  an  offeror  might  protest  the  contract  award?  * 

Please  choose  only  one  of  the  following: 

0  1  Not  at  all  concerned 


O  2 

Q  3 


0  4  Concerned 


0  5 
0  6 


0  7  Extremely  concerned 


ittps://su  rvey.nps.edu/admin/printaMesurvey /sa/index/surveyld/ 714497 
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NPS  Enterprise  Survey  -  Bid  Protest 


5/12/14,  10:34  PM 


Group  3 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  Disagree”  and  7  represents  "Strongly  Agree",  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  Disagree"  and  7  represents 
"Strongly  Agree",  rate  the  following  statements. 

I  was  empowered  to  make  required  decisions  throughout  the  source  selection. 

* 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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5/12/14,  10:34  PM 


[]I  was  trusted  that  the  decisions  I  made  throughout  the  source  selection  would  be 
appropriate.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

My  management  supported  me  on  the  decisions  I  made  during  the  source  selection. 

* 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[] 

If  I  disagreed  with  an  aspect  of  a  legal  opinion/review,  I  had  the  latitude  to  deviate 
from  it.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

I  had  to  change  documents  generated  during  the  source  selection  to  correspond 
with  reviewers.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[] 

I  might  as  well  not  have  a  warrant  since  my  decisions  were  overridden  by 
reviewers.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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Group  5 


On  a  scale  of  1  to  7,  where  1  represents  “Strongly  Disagree'  and  7  represents  “Strongly  Agree',  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  Disagree"  and  7  represents 
"Strongly  Agree",  rate  the  following  statements. 

The  milestones  for  awarding  this  contract  were  too  aggressive.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[]I  was  not  rushed  to  award  this  contract.  * 

Please  choose  only  one  of  the  following: 

0  1 :  Strongly  disagree 
0  2:  Disagree 
O  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
O  6:  Agree 
0  7:  Strongly  agree 


[] 

I  had  sufficient  time  to  get  this  contract  awarded.  * 

Please  choose  only  one  of  the  following: 

0  1 :  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[]At  least  once,  a  technical  evaluator  was  required  to  change  the  wording  of  their 
technical  evaluations.  * 

Please  choose  only  one  of  the  following: 

O  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

At  least  one  technical  evaluator  expressed  concern  about  not  being  able  to  say  what 
needs  to  be  said  in  the  technical  evaluation.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[] 

At  least  one  technical  evaluator  was  concerned  that  the  constraints  imposed  on 
their  evaluations  impeded  his/her  ability  to  write  a  meaningful  evaluation.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
O  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

The  technical  evaluators  believed  that  the  quality  of  their  evaluations  could  not 
have  been  better.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[] 

If  there  were  no  Federal  Acquisition  Regulations,  no  source  selection  policy,  and  no 
threat  of  a  bid  protest,  the  quality  of  the  technical  evaluations  would  have  been  the 
same.  * 

Please  choose  only  one  of  the  following : 

O  1:  Strongly  disagree 
O  2:  Disagree 
0  3:  Somewhat  disagree 
O  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


□  Upon  evaluation  of  proposals,  at  least  one  technical  evaluator  expressed  a  need  to 
change  at  least  one  evaluation  criterion  or  its  definition.  * 

Please  choose  only  one  of  the  following: 

O  1 :  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[]As  a  portion  of  the  customer's  total  annual  spending  amount,  the  dollar  value  of 
this  requirement  was  high.  * 

Please  choose  only  one  of  the  following: 

Q  1:  Strongly  disagree 
O  2:  Disagree 
O  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

This  requirement  was  important  for  the  good  operation  of  our  customer's 
organization.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
O  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
O  6:  Agree 
0  7:  Strongly  agree 
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[] 

This  requirement  supported  a  core  competency  of  our  customer's  organization.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

Compared  to  other  purchases  for  this  customer,  this  requirement  was  important.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[] 

An  unsuccessful  outcome  of  the  RFP  would  have  had  only  minor  consequences  to 
our  customer.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  disagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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Group  6a 

The  next  five  questions  refer  to  the  contractor  that  was  awarded  a  contract  resulting  from  the  selected  source  selection.  For  a 
multiple-award  IDIQ,  select  the  contractor  that  received  the  first  delivery/task  order  award  and  answer  with  respect  to  that 
delrvery/task  order. 

On  a  scale  of  1  to  7,  where  1  represents  “Strongly  Disagree”  and  7  represents  “Strongly  Agree",  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


The  next  five  questions  refer  to  the  contractor  that  was  awarded  a  contract 
resulting  from  the  selected  source  selection.  For  a  multiple-award  IDIQ,  select  the 
contractor  that  received  the  first  delivery/task  order  award  and  answer  with 
respect  to  that  delivery/task  order.  On  a  scale  of  1  to  7,  where  1  represents 
"Strongly  Disagree"  and  7  represents  "Strongly  Agree",  rate  the  following 
statements. 


Our  customer  regrets  the  decision  to  do  business  with  this  contractor.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
O  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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□  Overall,  our  customer  is  very  satisfied  with  this  contractor.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
Q  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

Our  customer  is  very  pleased  with  what  the  contractor  does  for  them.  * 

Please  choose  only  one  of  the  following: 

0  1 :  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[] 

Our  customer  is  not  completely  happy  with  this  contractor.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

If  we  had  to  do  it  all  over  again,  we  would  still  choose  to  use  this  contractor.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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Group  6b 


The  next  seven  questions  refer  to  the  contractor  and  contract  or  delivery/task  order  used  to  answer  the  previous  five  questions.  On 
a  scale  of  1  to  7,  where  1  represents  "needs  improvement”  and  7  represents  “superior  performance”,  rate  the  following. 


[] 

The  next  seven  questions  refer  to  the  contractor  and  contract  or  delivery/task  order 
used  to  answer  the  previous  five  questions.  On  a  scale  of  1  to  7,  where  1 
represents  "needs  improvement"  and  7  represents  "superior  performance",  rate  the 
following. 


Product/service  quality  per  specifications  * 

Please  choose  only  one  of  the  following: 

O  1  Needs  improvement 

O  2 

O  3 

0  4  Adequate  performance 

O  5 

Q  6 

0  7  Superior  Performance 


□  Delivery  performance  per  specifications  * 

Please  choose  only  one  of  the  following: 

0  1  Needs  improvement 
0  2 
0  3 

0  4  Adequate  performance 

O  5 

O  6 

0  7  Superior  Performance 
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□  Product/service  consistently  meets  customer  expectations  * 

Please  choose  only  one  of  the  following: 

0  1  Needs  improvement 

0  2 

O  3 

0  4  Adequate  performance 

O  5 

O  6 

0  7  Superior  Performance 


[] 

Responsiveness  to  requests  for  changes  * 

Please  choose  only  one  of  the  following: 

0  1  Needs  improvement 

O  2 

O  3 

0  4  Adequate  performance 

0  5 
O  6 

0  7  Superior  Performance 
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[]Required  service  and/or  technical  support  * 

Please  choose  only  one  of  the  following: 

0  1  Needs  improvement 
0  2 
0  3 

0  4  Adequate  performance 

O  5 

O  6 

0  7  Superior  Performance 


[] Non-conformance  rate  * 
Please  choose  only  one  of  the  following: 

0  1  Needs  improvement 
0  2 
0  3 

0  4  Adequate  performance 

0  5 
0  6 

0  7  Superior  Performance 


□  Overall  performance  * 

Please  choose  only  one  of  the  following: 
0  1  Needs  improvement 

O  2 

O  3 

0  4  Adequate  performance 

O  5 

0  6 

0  7  Superior  Performance 
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Group  7 

On  a  scale  of  -5  to  5,  where  -5  represents  “completely  undesirable’  and  5  represents  “completely  desirable,"  please  rate  the  extent 
of  desirability  associated  with  each  possible  consequence  of  a  bid  protest. 


[] 

On  a  scale  of  -5  to  5,  where  -5  represents  "completely  undesirable"  and  5 
represents  "completely  desirable,"  please  rate  the  extent  of  desirability  associated 
with  each  possible  consequence  of  a  bid  protest. 


Increased  costs  to  settle  a  terminated  contract(s).  * 

Please  choose  only  one  of  the  following: 

0  -5  Completely  undesirable 

□  -4 
0  -3 
O  -2 
O  -i 

0  0  Does  not  matter 

O  1 

O  2 

O  3 

O  4 

0  5  Completely  desirable 
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[]Time  delay  to  the  mission.  * 

Please  choose  only  one  of  the  following: 

C)  -5  Completely  undesirable 

O  -4 
D  -3 

O  -2 
O  -1 

O  0  Does  not  matter 

O  1 

O  2 

O  3 

O  4 

O  5  Completely  desirable 


[] 

Enibarrassment/shame.  * 

Please  choose  only  one  of  the  following: 

O  -5  Completely  undesirable 

O  -4 
O  -3 

Q  -2 

O  -i 

O  0  Does  not  matter 

O  1 

D  2 

D  3 

O  4 

O  5  Completely  desirable 
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□increase  in  workload  to  resolve  the  protest.  * 

Please  choose  only  one  of  the  following: 

0  -5  Completely  undesirable 
0  -4 
O  -3 

O  -2 
O  -1 

O  0  Does  not  matter 

O  1 

O  2 

O  3 

0  4 

0  5  Completely  desirable 


[]Career  repercussions  for  making  a  mistake  or  omission  that  caused  a  bid  protest. 

* 

Please  choose  only  one  of  the  following: 

0  -5  Completely  undesirable 

O  -4 
O  -3 

O  -2 

0  -1 

0  0  Does  not  matter 
0  1 

0  2 
0  3 

O  4 

0  5  Completely  desirable 
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Group  8 

In  the  event  of  a  bid  protest  please  rate  the  probability  that  each  consequence  might  occur. 


[] 

In  the  event  of  a  bid  protest  please  rate  the  probability  that  each  consequence 
might  occur. 

Increased  costs  to  settle  a  terminated  contract(s).  * 

Please  choose  only  one  of  the  following: 

O  0% 

O  10% 

Q  20% 

O  30% 

O  40% 

O  50% 

O  60% 

O  70% 

O  80% 

Q  90% 

O  ioo% 
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[]Tinie  delay  to  the  mission.  * 

Please  choose  only  one  of  the  following: 

Q 

0% 

Q 

10% 

D 

20% 

O 

30% 

O 

40% 

O 

50% 

o 

60% 

o 

70% 

o 

80% 

o 

90% 

o 

100% 

[]Embarrassment/shame.  * 

Please  choose  only  one  of  the  following: 

o  0% 

Q  io% 

O  20% 

O  30% 

O  40% 

O  50% 

O  60% 

O  70% 

O  80% 

Q  90% 

□  100% 
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Group  9 

On  a  scale  of  1  to  7,  where  1  represents  “Strongly  Disagree”  and  7  represents  ‘Strongly  Agree”,  rate  the  following  statements. 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


On  a  scale  of  1  to  7,  where  1  represents  "Strongly  Disagree"  and  7  represents 
"Strongly  Agree",  rate  the  following  statements. 

Our  acquisition  strategy  was  the  best  means  to  source  our  requirement.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[]Our  acquisition  strategy  was  the  best  means  to  achieve  our  acquisition  objectives. 

* 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

It  would  have  been  difficult  to  achieve  our  goals  without  the  use  of  our  acquisition 
strategy.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[] 

The  source  selection  method  we  used  (i.e.,  Low-Price-Technically  Acceptable;  full- 
tradeoff;  or  Price-Past  Performance  Tradeoff)  was  the  most  appropriate  for  this 
requirement.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
O  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 


[] 

Our  acquisition  strategy  ensured  we  selected  the  optimal  offeror.  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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[] 

Our  acquisition  strategy  provided  the  best  fit  to  the  buying  situation  (e.g., 
complexity,  dollar  value,  acquisition  objectives,  contract  length,  performance  risk, 
criticality  to  the  mission,  availability  of  supply,  time  available  to  award  a  contract, 

etc.)-  * 

Please  choose  only  one  of  the  following: 

0  1:  Strongly  disagree 
0  2:  Disagree 
0  3:  Somewhat  disagree 
0  4:  Neither  diagree  or  agree 
0  5:  Somewhat  agree 
0  6:  Agree 
0  7:  Strongly  agree 
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Group  10 
[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 

During  acquisition  planning,  how  many  days  were  planned  in  the  milestones  from 
receipt  of  a  complete  requirements  package  to  award  of  the  contract(s):  * 

Please  write  your  answer  here: 


[]Number  of  actual  days  from  receipt  of  a  complete  requirements  package  to  award 
of  the  contract  * 

Please  write  your  answer  here: 


□  Number  of  people  on  the  source  selection  team  including  all  advisors,  reviewers, 
full-time  participants,  and  part-time  participants:  * 

Please  write  your  answer  here: 


□Absent  a  risk  of  a  bid  protest,  in  your  opinion,  how  many  people  ideally  should 
have  been  on  the  source  selection  team:  * 

Please  write  your  answer  here: 
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[]What  source  selection  method  was  used?  (Select  one.)  * 

Please  choose  only  one  of  the  following: 

0  Low-Price-Technically-Acceptable 
0  Price-Past  Performance  Tradeoff 
0  Full  tradeoff 


□ 

Rate  the  appropriateness  of  each  source  selection  method  for  the  requirement  on  a 
scale  of  1  to  7,  where  1  represents  "completely  inappropriate"  and  7  represents 
"completely  appropriate."  Appropriate,  in  this  context,  means  the  source  selection 
method  is  the  best  fit  to  the  buying  situation  (e.g.,  complexity,  dollar  value, 
acquisition  objectives,  contract  length,  performance  risk,  criticality  to  the  mission, 
availability  of  supply,  time  available  to  award  a  contract,  etc.) 


LPTA  (Lowest  Price  Technically  Acceptable)  * 

Please  choose  only  one  of  the  following: 

0  1:  Completely  inappropriate 

0  2: 

O  3: 

0  4: 

O  5: 

O  6: 

0  7:  Completely  appropriate 
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[]PPT  (Price-Past  Performance  Tradeoff  )  * 

Please  choose  only  one  of  the  following: 

Q  1:  Completely  inappropriate 

O  z 

O  3: 

O  4: 

O  5: 

O  6: 

O  7:  Completely  appropriate 


□  Full  Tradeoff  * 

Please  choose  only  one  of  the  following: 

O  1:  Completely  inappropriate 

O  z 

O  3: 

O  4: 

O  5: 

Q  6: 

O  7:  Completely  appropriate 
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[] 

On  a  scale  of  1  to  7,  where  1  represents  "completely  satisfied"  and  7  represents 
"completely  dissatisfied,"  rate  your  level  of  satisfaction  with  the  freedom  to  openly 
discuss  those  aspects  of  the  proposals  that  needed  to  be  discussed  with  the  offeror 
in  order  to  fully  understand  the  offer  and  to  properly  evaluate  the  proposals.  * 

Please  choose  only  one  of  the  following: 

0  1:  Completely  satisfied 
0  2: 

0  3: 

0  4: 

0  5: 

0  6: 

0  7:  Completely  dissatisfied 


□  Hypothetically  if  there  was  no  ability  to  protest,  rate  the  extent  that  discussions 
with  offerors  would  have  differed,  on  a  scale  of  1  to  7,  where  1  represents  "no 
difference,"  and  7  represents  "substantial  difference."  * 

Please  choose  only  one  of  the  following: 

0  1  No  difference 
0  2 
0  3 
0  4 

0  5 
0  6 

0  7  substantial  difference 


[]Did  the  RFP  advise  offerors  that  the  Government  intended  to  award  without 
discussions?  * 

Please  choose  only  one  of  the  following: 

0  Yes 
O  No 
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[]Did  you  conduct  discussions?  * 

Please  choose  only  one  of  the  following: 

0  Yes 
C>  No 


[] 

On  a  scale  of  1  to  7,  where  1  represents  "completely  inappropriate"  and  7 
represents  "completely  appropriate,"  considering  the  risk,  criticality,  dollar  value, 
contribution  to  the  mission,  and  complexity,  rate  the  appropriateness  of  awarding  a 
contract  without  conducting  discussions.  [If  you  intended  to  conduct  discussions, 
select  1].  * 

Please  choose  only  one  of  the  following: 

O  1:  Completely  inappropriate 

O  2: 

O  3: 

0  4: 

D  5: 

0  6: 

□  7:  Completely  appropriate 


[]Did  offerors  make  oral  presentations?  * 

Please  choose  only  one  of  the  following: 

0  Yes 
0  No 
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[] 

On  a  scale  of  1  to  7,  where  1  represents  "completely  inappropriate"  and  7 
represents  "completely  appropriate,"  considering  the  risk,  criticality,  dollar  value, 
contribution  to  the  mission,  and  complexity,  rate  the  appropriateness  of  not 
utilizing  oral  presentations.  [If  offerors  made  oral  presentations,  select  1].  * 

Please  choose  only  one  of  the  following: 

0  1:  Completely  inappropriate 

0  2: 

0  3: 

0  4: 

O  5: 

O  6: 

0  7:  Completely  appropriate 


□According  to  your  acquisition  planning,  how  many  contracts  were  originally 
intended  to  be  awarded?  * 

Please  write  your  answer  here: 


[]How  many  contracts  were  actually  awarded?  * 

Please  write  your  answer  here: 
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Group  10a 

During  the  source  selection,  how  many  iterations  of  each  of  the  following  documents  were  generated? 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 


During  the  source  selection,  how  many  iterations  of  each  of  the  following 
documents  were  generated? 

Source  selection  decision  document  * 

Please  write  your  answer  here: 


□Comparative  assessment/proposal  analysis  report  * 

Please  write  your  answer  here: 


□  Evaluation  notices  (sum  of  iterations  for  all  ENs  across  all  offerors)  * 

Please  write  your  answer  here: 
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□  Source  selection  plan  * 
Please  write  your  answer  here: 


□  Debriefing  scripts  (sum  of  iterations  for  all  scripts  across  all  offerors)  * 

Please  write  your  answer  here: 


□Technical  evaluations  (sum  of  iterations  for  all  evaluations  across  all  offerors)  * 

Please  write  your  answer  here: 


□  Past  performance  evaluations  (sum  of  iterations  for  all  evaluations  across  all 
offerors)  * 

Please  write  your  answer  here: 


□Cost/Price  analysis  * 
Please  write  your  answer  here: 


□  Color/rating  chart  * 

Please  write  your  answer  here: 
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[] 

Evaluation  briefing  charts  for  reviewers  and  SSA  * 

Please  write  your  answer  here: 
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Group  11 


[] 

Please  refer  back  to  the  most-recently  completed  FAR  Part  15  source  selection 
above  $150K  in  which  you  participated.  Answer  the  questions  in  this  survey 
pertaining  to  this  selected  source  selection. 

Was  the  contract  award  protested?  * 

Please  choose  only  one  of  the  following: 

O  Yes 

0  No 


[] 

What  was  the  contract  type?  * 

Please  choose  only  one  of  the  following: 

C)  Fixed  Price 
0  Cost  Reimbursement 
0  Time  and  Materials 
0  Labor-Hour 

0  Hybrid  (multiple  contract  types) 

0  Other 


[]What  was  the  total  dollar  value  of  the  contract/task  order/delivery  order 
(including  all  options)?  * 

Please  write  your  answer  here: 
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□  Was  the  requirement  set  aside  for  Small  Business?  * 

Please  choose  only  one  of  the  following: 

O  Yes 
O  No 


[] 

What  type  of  supply  or  service  is  the  contractor  providing?  * 

Please  choose  only  one  of  the  following: 

C>  Service 
O  Construction 

C>  Supplies/Commodities/Spares 
Q  Weapon  System 
O  Other  Capital  Equipment 


[]For  this  contract/task  order/delivery  order,  what  was  the  Product  Service 
Code/Federal  Supply  Class  code?  This  group  code  is  the  first  digit(s)  of  the 
PSC/FSC  code. 

Please  write  your  answer  here: 


https://survey.nps.edu/admin/printAMesurvey/u/index/vurveyid/ 714497 


Pape  45  ol  54 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  147- 


NPS  Enterprise  Survey  -  Bid  Protest 


S/12/14, 10:34  PM 


Group  12 


[] 

ALMOST  DONE!!! 


Please  refer  back  to  the  most  recently  completed  FAR  part  15  source  selection 
above  150K  in  which  you  participated.  Answer  the  questions  pertaining  to  this 
source  selection. 


For  the  following  roles  please  provide  the  number  of  people  within  that  role,  their 
Grade(s)/Rank(s),  and  the  fraction  of  1  year's  time  and  effort  dedicated  to  the 
Source  Selection.  If  no  involvement,  just  skip  to  the  next  role. 

Format:  (#  of  People;Grade;Fraction  of  one  year's  time) 


Example  #1 

For  example,  if  there  were  3  contracting  officers  involved,  and  a  GS  13  spent  3 
months  of  their  time  to  the  source  selection  and  two  GS  12's  spent  4  months  of 
their  time,  the  appropriate  response  would  be: 

(3;  GS13,GS12,GS12;  25%,33%,33%) 

Example  #2 

If  only  one  GS  12  was  involved  and  spent  6  months  on  the  source  selection,  the 
appropriate  response  would  be: 

(1;  GS12;  50%) 

Example  #3 

If  one  technical  evaluator  (GS12)  allocated  half  of  his/her  time  to  the  source 
selection  for  6  months,  insert  25%  [=  (.5  year)*(.5  time/effort)].  The  appropriate 
response  would  be: 


(1;  GS12;  25%) 
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CONTRACTING  OFFICER 

Please  write  your  answer  here: 


□  CONTRACT  SPECIALISTS 

Please  write  your  answer  here: 


□TECHNICAL  EVALUATORS 

Please  write  your  answer  here: 


□  COST/  PRICE  ANALYSTS 

Please  write  your  answer  here: 


□supervisors 

Please  write  your  answer  here: 


□legal 

Please  write  your  answer  here: 


https://surwy.nps.edu/admin/printAbiesurvey/sarindex/surveyid/714497 
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[]PAST  PERFORMANCE  TEAM 

Please  write  your  answer  here: 


[] 

SOURCE  SELECTION  ADVISORY  COUNCIL/NON-LEGAL  ADVISORS 

Please  write  your  answer  here: 


[] 

SOURCE  SELECTION  AUTHORITY 


Please  write  your  answer  here: 


[] 

FINANCIAL  MANAGEMENT 

Please  write  your  answer  here: 


[] 

PROGRAM  MANAGER/REQUIREMENTS  OFFICE 

Please  write  your  answer  here: 


https://survey.nps.edU/admin/printablesurvey/u/index/surveyid/714497 
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NPS  Enterprise  Survey  -  Bid  Protect 


5/12/14,  10:34  PM 


[] 

SMALL  BUSINESS  REPRESENTATIVE 

Please  write  your  answer  here: 


□  CONTRACTED  CONSULTANTS/LABOR 

Please  write  your  answer  here: 


[] 

Please  write  your  answers)  here: 

OTHER  (DESCRIPTION) 

(#  of  People;Grade;Fractlon  of  one  year's 
time) 


httpv//Mjrvey.npi.edu/ad«nin/print.iblesurvey/u/index/iurveyid/ 714497 
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Group  13 

DEMOGRAPHICS 
LAST  PAGE!!!! 


[] 

LAST  PAGE!! 


General  questions  and  demographics.  Please  answer  the  following  questions 
accordingly. 


Throughout  your  career,  how  many  times  have  you  awarded  a  task  order/delivery 
order  against  an  IDIQ  contract  (or  a  call  against  a  blanket  purchase  agreement)  in 
order  to  avoid  a  protestable  competitive  procurement?  * 

Please  write  your  answer  here: 


[] 

Throughout  your  career,  how  many  times  have  you  sole  sourced  a  requirement  in 
order  to  avoid  a  protestable  competitive  procurement?  * 

Please  write  your  answer  here: 


□Throughout  your  career,  how  many  times  have  you  found  a  way  to  modify  an 
existing  contract,  order,  call,  or  agreement  in  order  to  avoid  a  protestable 
competitive  procurement?  * 

Please  write  your  answer  here: 


httpv  /  lux  rvey.npi.edu/admin/ ptinuMesurvey/u /index/surveyid/ 714  497 
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□Throughout  your  career,  how  many  bid  protests  have  you  experienced?  * 

Please  write  your  answer  here: 


□Throughout  your  career,  have  you  ever  experienced  a  sustained  bid  protest?  * 

Please  choose  only  one  of  the  following: 

0  Yes 

0  No 


□ 

How  many  total  years  of  experience  do  you  have  in  contracting?  * 

Please  write  your  answer  here: 


[] 

In  how  many  source  selections  have  you  participated  throughout  your  career? 

* 

Please  write  your  answer  here: 


https://survey.nps.edu/admin/pfintaMesurv'ey/sa/index/surveyid/714497 
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□  What  is  the  highest  level  of  Acquisition  Professional  Development  Program 
(APDP)  certification  that  you  hold?  * 

Please  choose  only  one  of  the  following: 

0  APDP/DAWIA  Level  1 
O  APDP/DAWIA  Level  2 
0  APDP/DAWIA  Level  3 
0  No  APDP  or  DAWIA  Certification 


[] 

What  is  the  highest  level  of  education  that  you  have  attained?  * 

Please  choose  only  one  of  the  following: 

0  High  School  Diploma  /  GED 
0  Associates 
0  Bachelors 
0  Masters 

0  Doctoral  /  Professional 


[]How  many  years  of  experience  do  you  have  evaluating  contractor  performance?  * 

Please  write  your  answer  here: 


[] 

What  is  your  gender? 

Please  choose  only  one  of  the  following: 

0  Female 
0  Male 


http*:  /  /  uirvey.nps.edu/admin/  pr  intaMesurvey >'sa  /index/surveyid/  7144  97 


Page  52  of  54 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  154- 


NPS  Enterprise  Survey  -  Bid  Protest 
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[] 

We  appreciate  any  comments  or  feedback  you  can  provide  on  the  topic  of  contractor 
protests  in  government  contracting  and/or  this  survey,  (optional) 

Please  write  your  answer  here: 


https:  /  /su  rvey.nps.edu/admin/  print&Mesurvey/sa  ,'index/surveyid  /  714497 
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APPENDIX  B. 


SURVEY  INVITATIONS 


NAVAL 

POSTGRADUATE 

SCHOOL 


Dear  Colleague, 

You  have  been  randomly  selected  to  participate  in  a  study  of  bid  Protests. 
This  research  is  approved  by  Mr.  Elliot  Branch,  Deputy  Assistant  Secretary  of  the 
Navy  for  Acquisition  and  Procurement,  and  will  help  us  fulfill  graduation 
requirements  for  our  MBA  degree. 

The  purpose  of  this  study  is  to  explore  the  effects  of  potential  bid  protests  on 
acquisition  strategies.  I  respectfully  request  your  assistance  to  complete  the  web- 
based  survey  located  at  the  hyperlink  below.  Your  participation  is  completely 
anonymous. 

For  your  time,  you  will  be  eligible  to  enter  a  random  drawing  for  a  new 
iPad  Mini,  16  GB.  To  enter,  follow  the  instructions  at  the  end  of  the  survey. 

Survey  Link:  https://survey.nps.edu/714497/lanq-en 

Your  participation  is  voluntary;  however,  responses  are  vital  to  conducting 
valid  research  that  represents  your  knowledge  and  experience.  Please  complete  the 
survey  no  later  than  15  FEB  2014.  The  survey  should  take  no  longer  than  30 
minutes  to  complete. 

We  very  much  appreciate  any  assistance  you  can  provide.  We  would  be 
more  than  happy  to  share  our  findings  with  you  once  we  complete  our  research.  If 
you  have  any  questions  or  comments,  please  feel  free  to  contact  myself  at 
jdcaland@nps.edu  ,  or  my  Principle  Advisor,  Lt.  Col.  Timothy  Hawkins,  PH.D.,  USAF 
at  timothy.hawkins@wku.edu  .  Questions  about  your  rights  as  a  research  subject 
may  be  addressed  to  the  Navy  Postgraduate  School  IRB  Chair,  Dr.  Larry  Shattuck, 
831-656-2473,  lqshattu@nps.edu. 

V/R 

LCDR  Suquon  Combs,  LT  Jason  Calandruccio,  LT  Brian  Colbert 
MBA  Students,  Naval  Postgraduate  School 

Principle  Advisor:  Lt.  Col.  Timothy  Hawkins,  PH.D.,  USAF 
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Advisor:  E.  Cory  Yoder,  Senior  Lecturer,  Naval  Postgraduate  School. 

******************************************************************************* 

This  research  is  being  conducted  through  the  Naval  Postgraduate  School’s 
Graduate  School  of  Business  and  Public  Policy.  The  questionnaire  is  anonymous; 
your  responses  cannot  be  linked  to  you.  There  are  not  necessarily  “right  answers.” 

Procedures.  Your  extent  of  participation  in  this  research  involves  only  the  completion 
of  this  questionnaire. 

Synopsis.  This  is  both  an  anonymous  and  voluntary  questionnaire.  (Please  note,  in 
order  to  obtain  consistent  and  usable  results,  it  is  important  that  you  answer  all 
questions).  It  will  take  most  respondents  approximately  30  minutes  to  complete  the 
questionnaire. 

Risks  and  Benefits.  Your  participation  in  this  research  poses  no  known  risk.  You  will 
be  asked  questions  pertaining  to  the  latest  source  selection  in  which  you 
participated.  There  will  be  no  personal  benefits  beyond  having  contributed  your 
expertise  to  this  important  research.  Results  of  the  survey  will  be  used  responsibly 
and  protected  against  release  to  unauthorized  persons;  however,  there  is  minor  risk 
that  data  collected  could  be  mismanaged.  If  desired,  you  may  contact  the  researcher 
above  if  you  would  like  to  receive  a  report  of  the  results  of  the  study. 

Confidentiality  and  Privacy  Act.  All  records  of  this  study  will  be  kept  confidential  and, 
since  responses  are  anonymous,  your  privacy  will  not  be  at  risk.  No  information  will 
be  publicly  accessible  which  could  identify  you  as  a  participant.  Responses  will  be 
maintained  by  NPS  for  ten  years,  after  which  they  will  be  destroyed. 

By  taking  this  survey,  I  am  acknowledging  that  I  have  read  and  understand  this 
information,  that  I  understand  the  nature  and  purpose  of  this  study,  including  its  risks 
and  benefits,  and  that  I  agree  to  voluntarily  participate  in  this  online  survey.  I  also 
understand  that  I  may  discontinue  at  any  time  simply  by  exiting  this  website. 

Survey  Link:  https://survev.nps.edu/714497/lanq-en 
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Dear  Sir  or  Ma’am, 

We  recently  invited  you  to  participate  in  a  research  study  of  bid  protest 
conducted  by  the  Naval  Postgraduate  School.  We  regret  to  inform  you  that  a  raffle 
for  an  iPad  Mini  will  not  occur  due  to  ethics  regulation  constraints  offered  by 
supplemental  legal  advice.  The  research  will  proceed,  and  if  you  have  not  had  a 
chance  to  participate,  please  consider  offering  your  expertise. 

Survey  Link:  https://survey.nps.edu/714497/lang-en 

Thank  you  and  we  apologize  for  any  inconvenience. 

V/R 

LCDR  Suquon  Combs,  LT  Jason  Calandruccio,  LT  Brian  Colbert 
MBA  Students,  Naval  Postgraduate  School 

Principle  Advisor:  Lt.  Col.  Timothy  Hawkins,  PH.D.,  USAF 
Advisor:  E.  Cory  Yoder,  Senior  Lecturer,  Naval  Postgraduate  School. 
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Dear  Colleague, 

Thank  you  for  your  participating  in  our  research  study.  We  need  several  more 
responses  to  achieve  the  required  response  rate.  For  those  of  you  who  have  not 
been  able  to  respond  just  yet,  please  take  a  few  minutes  and  complete  the  survey. 
For  those  of  you,  who  have  already  responded,  thank  you  much! 

If  you  have  not  served  as  a  Contracting  officer  or  contract  specialist  on  a  FAR 
Part  15  source  selection,  please  reply  and  let  me  know.  I  will  remove  you  from  the 
list.  This  will  improve  the  accuracy  of  our  response  rate. 

As  a  reminder,  we  are  conducting  a  study  examining  the  effects  of  potential 
bid  protests.  In  appreciation  of  your  participation,  I  will  be  happy  to  send  you  a  brief 
of  the  study  results.  Just  reply  to  this  message  to  request  the  summary  report. 
Again,  this  report  will  contain  descriptive  statistics  based  on  collective  responses  of 
all  participating  organizations;  no  individual  response  data  will  be  published.  Your 
response  will  be  completely  anonymous.  None  of  the  information  put  in  the  survey 
can  be  traced  back  to  any  individual  nor  to  any  organization.  Thank  you  for  assisting 
us  in  this  valuable  study  required  for  the  completion  of  my  degree  at  the  Naval 
Postgraduate  School. 

To  access  the  online  survey,  please  click  the  following  link: 
https://survey.nps.edu/714497/lang-en  (or  copy  and  paste  to  your  Web  browser.) 


V/R 

LT  Jason  Calandruccio 
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NAVAL 

POSTGRADUATE 

SCHOOL 

Dear  Colleague, 

We  apologize  for  not  providing  a  link  in  our  previous  e-mail.  Your  support  is 
greatly  appreciated. 

To  access  the  online  survey,  please  click  the  following  link: 
https://survey.nps.edU/7 1 4497/lanq-en  (or  copy  and  paste  to  your  Web  browser.) 

We  still  require  several  more  responses  for  our  survey  to  achieve  a  desired 
response  rate  for  our  Deputy  Assistant  Secretary  of  the  Navy  (Acquisition  and 
Procurement)  sponsored  Naval  Postgraduate  School  MBA  Graduate  project.  If  you 
have  not  had  an  opportunity  yet,  it  is  not  too  late.  The  survey  has  been  extended  to 
March  1 1 .  It  takes  about  30  minutes  and  will  not  only  aid  in  our  graduate 
requirements,  but  will  further  our  collective  understanding  of  our  contracting 
workforce.  If  you  would  like  a  copy  of  our  research,  we  would  be  more  than  happy  to 
provide  this  to  you  once  completed.  Just  send  an  e-mail  to  idcaland@nps.edu  or 
reply  to  this  e-mail  and  we  will  ensure  you  are  on  the  list  to  receive  it.  For  those  of 
you,  who  have  already  responded,  thank  you  much! 

If  you  will  not  be  able  to  respond,  please  let  us  know  why.  This  may  make  our 
response  rate  more  accurate.  Also,  please  inform  us  if  you  are  not  a  Contracting 
officer  or  specialist  with  experience  in  at  least  one  FAR  Part  15  source  selection;  we 
need  to  remove  you  from  the  distribution  list. 

Very  Respectfully, 

LCDR  Suquon  Combs,  SC,  USN 

LT  Jason  Calandruccio,  SC,  USN 

LT  Brian  Colbert,  SC,  USN 
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APPENDIX  C.  ENDORSEMENT  BY  DASN  (AP) 


DEPARTMENT  OF  THE  NAVY 

OFFICE  OF  THE  ASSISTANT  SECRETARY 
(RESEARCH,  DEVELOPMENT  AND  ACQUISITION) 
1000  NAVY  PENTAGON 
WASHINGTON  DC  20350-1000 


JUL  1  9  2013 


FIRST  ENDORSEMENT  on  Naval  Postgraduate  School  ltr  of  06  Jun  2013 

From:  Deputy  Assistant  Secretary  of  the  Navy  Acquisition  and  Procurement 
To:  Navy  Personnel  Research,  Studies,  and  Technology  (NPRST)  (PERS-14) 

Subj:  REQUEST  FOR  APPROVAL  OF  NAVY  PERSONNEL  SURVEY 

1.  Forwarded,  recommending  approval. 


Elliott  B  Branch 

Deputy  Assistant  Secretary  of  the  Navy 


Acquisition  and  Procurement 
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APPENDIX  D.  REGRESSION  OUTPUT  FROM  SPSS  VER  22 


Regression  ANTECEDENTS  OF  FEAR 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Number  of 

Source 
Selections, 
Protest  Risk, 
Dollar  Value, 
Requirement 
Criticality, 

PALT 

Planned, 

Years  of 
Experience13 

Enter 

a.  Dependent  Variable:  FEAR 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.36 1 a 

.131 

.115 

7.319 

Predictors:  (Constant),  Number  of  Source  Selections,  Protest  Risk,  Dollar 
Value,  Requirement  Criticality,  PALT  Planned,  Years  of  Experience 
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ANOVAa 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

2759.668 

6 

459.945 

8.587 

.000° 

Residual 

18373.021 

343 

53.566 

Total 

21132.689 

349 

a.  Dependent  Variable:  FEAR 

b.  Predictors:  (Constant),  Number  of  Source  Selections,  Protest  Risk,  Dollar  Value,  Requirement 
Criticality,  PALT  Planned,  Years  of  Experience 


Coefficients:  1 


Unstandardized 

Coefficients 

Standardized 

Coefficients 

Constructa 

B 

Std.  Error 

Beta 

t 

Sig. 

(Constant) 

10.995 

2.852 

3.856 

.000 

PALT  Planned 

-.259 

.091 

-.150 

-2.862 

.004 

Dollar  Value 

.721 

.238 

.155 

3.032 

.003 

Requirement 

Criticality 

.431 

.135 

.167 

3.183 

.002 

Protest  Risk 

.374 

.171 

.116 

2.190 

.029 

Years  of 

Experience 

.037 

.042 

.049 

.881 

.379 

Number  of  Source 
Selections 

-.267 

.104 

-.142 

-2.558 

.011 

a.  Dependent  Variable:  FEAR 


Regression  FEAR  TO  TEE 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR 

Stepwise 
(Criteria: 
Probability-of- 
F-to-enter  <= 
.050, 

Probability-of- 
F-to-remove 
>=  .100). 

a.  Dependent  Variable:  Quality  of  Evaluation  Factors 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.253a 

.064 

.061 

4.046 

a.  Predictors:  (Constant),  FEAR 


ANOVA3 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

390.783 

1 

390.783 

23.869 

,000D 

Residual 

5697.434 

348 

16.372 

Total 

6088.217 

349 

a.  Dependent  Variable:  Technical  Evaluation  Effectiveness 

b.  Predictors:  (Constant),  FEAR 


Coefficients3 


Unstandardized  Coefficients 

Standardized 

Coefficients 

Model 

B 

Std.  Error 

Beta 

t 

Sig. 

1  (Constant) 

16.343 

.590 

27.719 

.000 

FEAR 

-.136 

.028 

-.253 

-4.886 

.000 

a.  Dependent  Variable:  Quality  of  Evaluation  Factors 
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Logistic  Regression  FEAR  to  PALT  Actual 


Case  Processing  Summary 


Unweighted  Cases3 

N 

Percent 

Selected  Cases  Included  in 

Analysis 

350 

100.0 

Missing  Cases 

0 

.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total  number  of 
cases. 


Dependent  Variable 
Encoding 


Original  Value 

Internal  Value 

.000 

0 

1.000 

1 

Block  0:  Beginning  Block 


Classifica 

tion  Tablea,t> 

Predicted 

PALT  ACTUAL 

Percentage 

Observed 

.000 

1.000 

Correct 

Step  0 

PALT_ACTUAL  .000 

0 

145 

.0 

1.000 

0 

205 

100.0 

Overall  Percentage 

58.6 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  168- 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

.346 

.109 

10.184 

1 

.001 

1.414 

Variables  not  in  the  Equation 

Score 

df 

Sig. 

Step  0 

Variables 

FEAR 

12.318 

1 

.000 

Overall  Statistics 

12.318 

1 

.000 

Block  1:  Method  =  Enter 

Omnibus  Tests  of  Model  Coefficients 


Chi-square 

df 

Sig. 

Step  1  Step 

12.454 

1 

.000 

Block 

12.454 

1 

.000 

Model 

12.454 

1 

.000 

Model  Summary 


Step 

-2  Log 
likelihood 

Cox  &  Snell  R 
Square 

Nagelkerke  R 
Square 

1 

462.4123 

.035 

.047 

a.  Estimation  terminated  at  iteration  number  3  because  parameter 
estimates  changed  by  less  than  .001 . 


Hosmer  and  Lemeshow  Test 


Step 

Chi-square 

df 

Sig. 

1 

11.607 

8 

.170 
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Classification  Table1 


Predicted 

PALT_ACTUAL 

Percentage 

Observed 

.000 

1.000 

Correct 

Step  1 

PALT_ACTUAL  .000 

39 

106 

26.9 

1.000 

35 

170 

82.9 

Overall  Percentage 

59.7 

a.  The  cut  value  is  .500 


Contingency  Table  for  Hosmer  and  Lemeshow  Test 


PALTACT 

UAL  =  .000 

1 — 
o 
< 

1- 

—I 

< 

CL 

JAL  =  1.000 

Total 

Observed 

Expected 

Observed 

Expected 

Step  1  1 

22 

19.683 

12 

14.317 

34 

2 

17 

20.943 

23 

19.057 

40 

3 

20 

18.313 

18 

19.687 

38 

4 

17 

15.991 

19 

20.009 

36 

5 

10 

16.286 

30 

23.714 

40 

6 

17 

14.323 

21 

23.677 

38 

7 

14 

13.845 

25 

25.155 

39 

8 

14 

10.341 

17 

20.659 

31 

9 

12 

10.715 

24 

25.285 

36 

10 

2 

4.561 

16 

13.439 

18 

Variables  in  the  Equation 


B 


S.E. 


Wald 


df 


Sig. 


Exp(B) 


95%  C.I.for 
EXP(B) 


Lower 


Upper 


Step  FEAR 
1a  Constant 


.050 

-.630 


.014 

.300 


11.996 

4.419 


.001 

.036 


1.051 

.533 


1.022 


1.082 


a.  Variable(s)  entered  on  step  1 :  FEAR. 


Regression  FEAR  to  SSMA 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEARb 

Enter 

a.  Dependent  Variable:  SSMA 

b.  All  requested  variables  entered. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  170- 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of  the 

Estimate 

1 

.059a 

.003 

.001 

6.65306 

a.  Predictors:  (Constant),  FEAR 


ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

53.021 

1 

53.021 

1.198 

.275b 

Residual 

15403.596 

348 

44.263 

Total 

15456.617 

349 

a.  Dependent  Variable:  SSMA 

b.  Predictors:  (Constant),  FEAR 


Coefficients3 


Unstandardized  Coefficients 

Standardized 

Coefficients 

Model 

B 

Std.  Error 

Beta 

t 

Sig. 

1  (Constant) 

34.050 

.969 

35.123 

.000 

FEAR 

-.050 

.046 

-.059 

-1.094 

.275 

a.  Dependent  Variable:  SSMA 
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Logistic  Regression  FEAR  HI/LO  to  LPTA  Appropriateness 


Case  Processing  Summary 


Unweighted  Cases3 

N 

Percent 

Selected  Cases  Included  in  Analysis 

133 

38.0 

Missing  Cases 

217 

62.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total  number  of  cases. 


Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

Low 

0 

High 

1 

Block  0:  Beginning  Block 


Classification  Tablea,b 


Observed 

Predicted 

LPTA  Appropriateness 

Percentage 

Correct 

Low 

High 

Step  0  LPTA  Low 

120 

0 

100.0 

mduuruuridienebb 

High 

13 

0 

.0 

Overall  Percentage 

90.2 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 
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Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

-2.223 

.292 

57.939 

1 

.000 

.108 

Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR_LPTA_USED 

6.861 

1 

.009 

Overall  Statistics 

6.861 

1 

.009 

Block  1:  Method  =  Enter 

Omnibus  Tests  of  Model  Coefficients 


Chi-square 

df 

Sig. 

Step  1  Step 

6.559 

1 

.010 

Block 

6.559 

1 

.010 

Model 

6.559 

1 

.010 

Model  Summary 


Step 

-2  Log  likelihood 

Cox  &  Snell  R 
Square 

Nagelkerke  R 
Square 

1 

78.587a 

.048 

.102 

a.  Estimation  terminated  at  iteration  number  6  because  parameter 
estimates  changed  by  less  than  .001 . 

Classification  Table3 


Predicted 

LPTA  Appropriateness 

Percentage 

Observed 

Low 

High 

Correct 

Step  1  LPT A 

Inappropriateness 

Low 

120 

0 

100.0 

High 

13 

0 

.0 

Overall  Percentage 

90.2 

a.  The  cut  value  is  .500 
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Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  1a  FEAR_LPTA_USED 

1.542 

.632 

5.956 

1 

.015 

4.673 

Constant 

-3.008 

.512 

34.493 

1 

.000 

.049 

Logistic  Regression  FEAR  HI/LO  to  T/O  Appropriateness 


Case  Processing  Summary 


Unweighted  Cases3 

N 

Percent 

Selected  Cases  Included  in  Analysis 

174 

49.7 

Missing  Cases 

176 

50.3 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total  number  of  cases. 

Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

Appropriate 

0 

Not  Appropriate 

1 
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Block  0:  Beginning  Block 


Classification  Tablea,b 


Predicted 

TRADE-OFF  BINARY 

Percentage 

Observed 

Appropriate 

Not  Appropriate 

Correct 

Step  0  TRADE- 

Appropriate 

164 

0 

100.0 

OFF_BINARY 

Not 

Appropriate 

10 

0 

.0 

Overall  Percentage 

94.3 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

-2.797 

.326 

73.751 

1 

.000 

.061 

Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR_BINARY_TO 

1.835 

1 

.176 

Overall  Statistics 

1.835 

1 

.176 

Block  1:  Method  =  Enter 

Omnibus  Tests  of  Model  Coefficients 


Chi-square 

df 

Sig. 

Step  1  Step 

1.786 

1 

.181 

Block 

1.786 

1 

.181 

Model 

1.786 

1 

.181 
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Model  Summary 


Step 

-2  Log 
likelihood 

Cox  &  Snell  R 
Square 

Nagelkerke  R 
Square 

1 

74.758a 

.010 

.029 

a.  Estimation  terminated  at  iteration  number  6  because  parameter 
estimates  changed  by  less  than  .001. 


Hosmer  and  Lemeshow  Test 


Step 

Chi-square 

df 

Sig. 

1 

.000 

0 

Contingency  Table  for  Hosmer  and  Lemeshow  Test 


TRADE-OFF_BINARY  = 
Appropriate 

TRADE-OFF_BINARY  =  Not 
Appropriate 

Total 

Observed 

Expected 

Observed 

Expected 

Step  1  1 

101 

101.000 

4 

4.000 

105 

2 

63 

63.000 

6 

6.000 

69 

Classification  Table3 


Observed 

Predicted 

TRADE-OFF_BINARY 

Percentage 

Correct 

Appropriate 

Not 

Appropriate 

Step  1  TRADE-  Appropriate 

OFF  RIM  ARY 

164 

0 

100.0 

Urr  D 1  In  Mia  T 

Not  Appropriate 

10 

0 

.0 

Overall  Percentage 

94.3 

a.  The  cut  value  is  .500 
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Variables  in  the  Equation 


95%  C.I.for 
EXP(B) 

B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Lower 

Upper 

Step 

1a 

FEARBINARYTO 

-.877 

.665 

1.740 

1 

.187 

.416 

.113 

1.531 

Constant 

-2.351 

.427 

30.289 

1 

.000 

.095 

a.  Variable(s)  entered  on  step  1:  FEARBINARYTO. 


Regression  FEAR  HI/LO  to  Satisfaction  with  Discussions 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

HIGH_FEARb 

Enter 

a.  Dependent  Variable:  DISATDISC 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.1 1 1a 

.012 

.010 

1.710 

a.  Predictors:  (Constant),  HIGHFEAR 


ANOVAa 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

12.781 

1 

12.781 

4.370 

.037" 

Residual 

1017.779 

348 

2.925 

Total 

1030.560 

349 

a.  Dependent  Variable:  DISAT  DISC 

b.  Predictors:  (Constant),  HIGH  FEAR 
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Coefficients 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

3.087 

.130 

23.740 

.000 

HIGH_FEAR 

.382 

.183 

.111 

2.091 

.037 

a.  Dependent  Variable:  DISAT_DISC 


Regression  FEAR  to  Hypothetical  Change  in  Discussions 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

HIGH_FEARb 

Enter 

a.  Dependent  Variable:  if  there  was  no  ability  to  protest  question 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.31 6a 

.100 

.097 

1.93920 

a.  Predictors:  (Constant),  HIGH_FEAR 


AN  OVA3 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

145.309 

1 

145.309 

38.641 

,000D 

Residual 

1308.659 

348 

3.761 

Total 

1453.969 

349 

a.  Dependent  Variable:  if  there  was  no  ability  to  protest  question 

b.  Predictors:  (Constant),  HIGH_FEAR 
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Coefficients 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

2.723 

.147 

18.466 

.000 

HIGH_FEAR 

1.289 

.207 

.316 

6.216 

.000 

a.  Dependent  Variable:  if  there  was  no  ability  to  protest  question 


Logistic  Regression  FEAR  to  Discussions  Held  Y/N 


Case  Processing  Summary 


Unweighted  Cases3 

N 

Percent 

Selected  Cases  Included  in  Analysis 

350 

100.0 

Missing  Cases 

0 

.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total  number  of  cases. 


Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

No 

0 

Yes 

1 
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Block  0:  Beginning  Block 


Classification  Tablea,b 


Observed 

Predicted 

Discussions 

Percentage 

Correct 

No 

Yes 

Step  0  Discussions  No 

0 

147 

.0 

Yes 

0 

203 

100.0 

Overall  Percentage 

58.0 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

.323 

.108 

8.883 

1 

.003 

1.381 

Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR 

4.356 

1 

.037 

Overall  Statistics 

4.356 

1 

.037 

Block  1:  Method  =  Enter 


Omnibus  Tests  of  Model  Coefficients 


Chi-square 

df 

Sig. 

Step  1  Step 

4.370 

1 

.037 

Block 

4.370 

1 

.037 

Model 

4.370 

1 

.037 
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Model  Summary 


Step 

-2  Log 
likelihood 

Cox  &  Snell  R 
Square 

Nagelkerke  R 
Square 

1 

471.8343 

.012 

.017 

a.  Estimation  terminated  at  iteration  number  3  because  parameter 
estimates  changed  by  less  than  .001 . 


Classification  Table3 


Observed 

Predicted 

Discussions 

Percentage 

Correct 

No 

Yes 

Step  1  Discussions  No 

17 

130 

11.6 

Yes 

17 

186 

91.6 

Overall  Percentage 

58.0 

a.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  1a  FEAR 

.029 

.014 

4.315 

1 

.038 

1.030 

Constant 

-.251 

.295 

.722 

1 

.396 

.778 

a.  Variable(s)  entered  on  step  1:  FEAR. 


Regression  FEAR  to  Oral  Pres.  Appropriateness 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEARb 

Enter 

a.  Dependent  Variable:  ORAL  PRESENTATIONS 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.060a 

.004 

.001 

2.09783 

a.  Predictors:  (Constant),  FEAR 


ANOVAa 


Model 

Sum  of  Squares 

df 

Mean 

Square 

F 

Sig. 

1  Regression 

5.587 

1 

5.587 

1.269 

.261 b 

Residual 

1531.510 

348 

4.401 

Total 

1537.097 

349 

a.  Dependent  Variable:  ORALPRESENTATIONS 

b.  Predictors:  (Constant),  FEAR 


Coefficients3 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

5.023 

.306 

16.433 

.000 

FEAR 

-.016 

.014 

-.060 

-1.127 

.261 

a.  Dependent  Variable:  ORAL  PRESENTATIONS 

Regression  FEAR  to  Resources 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEARb 

Enter 

a.  Dependent  Variable:  NUMBER_OF_PEOPLE_ON_SS 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

,277a 

.077 

.074 

.28028 

a.  Predictors:  (Constant),  FEAR 


ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

2.280 

1 

2.280 

29.022 

,000b 

Residual 

27.338 

348 

.079 

Total 

29.618 

349 

a.  Dependent  Variable:  NUMBER_OF_PEOPLE_ON_SS 

b.  Predictors:  (Constant),  FEAR 


Coefficients3 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

.632 

.041 

15.476 

.000 

FEAR 

.010 

.002 

.277 

5.387 

.000 
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Regression  FEAR  Hi/Lo  to  Resources 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

HIGH_FEARb 

Enter 

a.  Dependent  Variable:  NUMBER_OF_PEOPLE_ON_SS 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

,257a 

.066 

.063 

.28197 

a.  Predictors:  (Constant),  HIGH_FEAR 


ANOVAa 


Model 

Sum  of 

Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

1.949 

1 

1.949 

24.513 

,000b 

Residual 

27.669 

348 

.080 

Total 

29.618 

349 

a.  Dependent  Variable:  NUMBER_OF_PEOPLE_ON_SS 

b.  Predictors:  (Constant),  HIGHFEAR 

Coefficients3 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

.761 

.021 

35.509 

.000 

HIGHFEAR 

.149 

.030 

.257 

4.951 

.000 

a.  Dependent  Variable:  NUMBER_OF_PEOPLE_ON_SS 
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Logistic  Regression  FEAR  to  More  Resources  than  needed 

Case  Processing  Summary 


Unweighted  Cases3 

N 

Percent 

Selected  Cases  Included  in  Analysis 

350 

100.0 

Missing  Cases 

0 

.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total  number  of  cases. 


Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

0 

0 

1 

1 

Block  0:  Beginning  Block 


Classification  Tablea,b 


Observed 

Predicted 

RESOURCES 

Percentage 

Correct 

0 

1 

Step  0  RESOURCES  0 

240 

0 

100.0 

1 

110 

0 

.0 

Overall  Percentage 

68.6 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

-.780 

.115 

45.909 

1 

.000 

.458 
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Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR 

.774 

1 

.379 

Overall  Statistics 

.774 

1 

.379 

Block  1:  Method  =  Enter 

Omnibus  Tests  of  Model  Coefficients 


Chi-square 

df 

Sig. 

Step  1  Step 

.776 

1 

.378 

Block 

.776 

1 

.378 

Model 

.776 

1 

.378 

Model  Summary 


Step 

-2  Log 
likelihood 

Cox  &  Snell  R 
Square 

Nagelkerke  R 
Square 

1 

434.965a 

.002 

.003 

a.  Estimation  terminated  at  iteration  number  3  because  parameter 
estimates  changed  by  less  than  .001 . 


Hosmer  and  Lemeshow  Test 


Step 

Chi-square 

df 

Sig. 

1 

11.643 

8 

.168 
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Contingency  Table  for  Hosmer  and  Lemeshow  Test 


RESOURCES  =  0 

RESOURCES  =  1 

Total 

Observed 

Expected 

Observed 

Expected 

Step  1  1 

28 

24.577 

6 

9.423 

34 

2 

27 

28.440 

13 

11.560 

40 

3 

29 

26.675 

9 

11.325 

38 

4 

23 

24.970 

13 

11.030 

36 

5 

25 

27.406 

15 

12.594 

40 

6 

21 

25.763 

17 

12.237 

38 

7 

23 

26.230 

16 

12.770 

39 

8 

25 

20.680 

6 

10.320 

31 

9 

26 

23.663 

10 

12.337 

36 

10 

13 

11.595 

5 

6.405 

18 

Classification  Table3 


Observed 

Predicted 

RESOURCES 

Percentage 

Correct 

0 

1 

Step  1  RESOURCES  0 

240 

0 

100.0 

1 

110 

0 

.0 

Overall  Percentage 

68.6 

a.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

95%  C.I.for 
EXP(B) 

Lower 

Upper 

Step  FEAR 

1a 

.013 

.015 

.773 

1 

.379 

1.013 

.984 

1.043 

Constant 

-1.040 

.319 

10.608 

1 

.001 

.353 

a.  Variable(s)  entered  on  step  1:  FEAR. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  187- 


Regression  FEAT  to  TCI 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEAR 

Stepwise 
(Criteria: 
Probability-of-F- 
to-enter  <=  .050, 
Probability-of-F- 
to-remove  >= 
.100). 

a.  Dependent  Variable:  TCI 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.31 8a 

.101 

.099 

.32938 

a.  Predictors:  (Constant),  FEAR 


ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

4.246 

1 

4.246 

39.135 

,000D 

Residual 

37.755 

348 

.108 

Total 

42.001 

349 

a.  Dependent  Variable:  TCI 

b.  Predictors:  (Constant),  FEAR 
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Coefficients 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

1.080 

.048 

22.505 

.000 

FEAR 

.014 

.002 

.318 

6.256 

.000 

a.  Dependent  Variable:  TCI 


Regression  FEAR  to  Source  Selection  Plan 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEARb 

Enter 

a.  Dependent  Variable:  TCI4 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

,290a 

.084 

.082 

.58979 

a.  Predictors:  (Constant),  FEAR 


AN  OVA3 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

11.147 

1 

11.147 

32.045 

,000b 

Residual 

121.051 

348 

.348 

Total 

132.198 

349 

a.  Dependent  Variable:  TCI4 

b.  Predictors:  (Constant),  FEAR 
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Coefficients3 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

1.110 

.086 

12.918 

.000 

FEAR 

.023 

.004 

.290 

5.661 

.000 

a.  Dependent  Variable:  TCI4 


Regression  FEAR  to  Debriefing  Script 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEARb 

Enter 

a.  Dependent  Variable:  TCI5 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

,280a 

.078 

.076 

.75232 

a.  Predictors:  (Constant),  FEAR 
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ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

16.707 

1 

16.707 

29.519 

,000b 

Residual 

196.964 

348 

.566 

Total 

213.672 

349 

a.  Dependent  Variable:  TCI5 

b.  Predictors:  (Constant),  FEAR 


Coefficients3 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

.729 

.110 

6.651 

.000 

FEAR 

.028 

.005 

.280 

5.433 

.000 

a.  Dependent  Variable:  TCI5 


Regression  FEAR  to  Color/Rating  Chart 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEARb 

Enter 

a.  Dependent  Variable:  TCI9 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.  1 95a 

.038 

.035 

.77260 

a.  Predictors:  (Constant),  FEAR 


AN  OVA3 


Model 

Sum  of 

Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

8.221 

1 

8.221 

13.772 

,000b 

Residual 

207.727 

348 

.597 

Total 

215.948 

349 

a.  Dependent  Variable:  TCI9 

b.  Predictors:  (Constant),  FEAR 


Coefficients3 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

.490 

.113 

4.350 

.000 

FEAR 

.020 

.005 

.195 

3.711 

.000 

a.  Dependent  Variable:  TCI9 
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Regression 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEARb 

Enter 

a.  Dependent  Variable:  TCI10 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.  1 96a 

.039 

.036 

.69098 

a.  Predictors:  (Constant),  FEAR 


ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

6.665 

1 

6.665 

13.959 

,000b 

Residual 

166.154 

348 

.477 

Total 

172.819 

349 

a.  Dependent  Variable:  TCI10 

b.  Predictors:  (Constant),  FEAR 
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Coefficients 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

.760 

.101 

7.548 

.000 

FEAR 

.018 

.005 

.196 

3.736 

.000 

a.  Dependent  Variable:  TCI10 

Regression  FEAR  to  TCP 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Fear  N=270b 

Enter 

a.  Dependent  Variable:  Transaction  Cost  Using  Sqrt 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

,143a 

.021 

.017 

.45284 

a.  Predictors:  (Constant),  Fear  N=270 


ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

1.155 

1 

1.155 

5.631 

,018b 

Residual 

54.957 

268 

.205 

Total 

56.112 

269 

a.  Dependent  Variable:  Transaction  Cost  Using  Sqrt 

b.  Predictors:  (Constant),  Fear  N=270 
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Coefficients 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

4.993 

.078 

63.838 

.000 

Fear  N=270 

.009 

.004 

.143 

2.373 

.018 

a.  Dependent  Variable:  Transaction  Cost  Using  Sqrt 


Logistic  Regression  FEAT  to  AC 

Case  Processing  Summary 


Unweighted  Cases3 

N 

Percent 

Selected  Cases  Included  in  Analysis 

350 

100.0 

Missing  Cases 

0 

.0 

Total 

350 

100.0 

Unselected  Cases 

0 

.0 

Total 

350 

100.0 

a.  If  weight  is  in  effect,  see  classification  table  for  the  total  number  of  cases. 


Dependent  Variable  Encoding 


Original  Value 

Internal  Value 

0 

0 

1 

1 
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Block  0:  Beginning  Block 


Classification  Tablea,b 


Predicted 

Binary. ..if  more  were  awarded 

Percentage 

Observed 

0 

1 

Correct 

Step  0  Binary.. .if  more  0 

331 

0 

100.0 

were  awarded  i 

19 

0 

.0 

Overall  Percentage 

94.6 

a.  Constant  is  included  in  the  model. 

b.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

Step  0  Constant 

-2.858 

.236 

146.737 

1 

.000 

.057 

Variables  not  in  the  Equation 


Score 

df 

Sig. 

Step  0  Variables  FEAR 

.863 

1 

.353 

Overall  Statistics 

.863 

1 

.353 

Block  1:  Method  =  Enter 

Omnibus  Tests  of  Model  Coefficients 


Chi-square 

df 

Sig. 

Step  1  Step 

.872 

1 

.350 

Block 

.872 

1 

.350 

Model 

.872 

1 

.350 
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Model  Summary 


Step 

-2  Log 
likelihood 

Cox  &  Snell  R 
Square 

Nagelkerke  R 
Square 

1 

146.7903 

.002 

.007 

a.  Estimation  terminated  at  iteration  number  6  because  parameter 
estimates  changed  by  less  than  .001. 


Hosmer  and  Lemeshow  Test 


Step 

Chi-square 

df 

Sig. 

1 

3.247 

8 

.918 

Contingency  Table  for  Hosmer  and  Lemeshow  Test 


Binary... if  more  were  awarded 
=  0 

Binary. ..if  more  were  awarded 
=  1 

Total 

Observed 

Expected 

Observed 

Expected 

Step  1  1 

33 

32.751 

1 

1.249 

34 

2 

38 

38.338 

2 

1.662 

40 

3 

37 

36.270 

1 

1.730 

38 

4 

34 

34.220 

2 

1.780 

36 

5 

36 

37.853 

4 

2.147 

40 

6 

36 

35.816 

2 

2.184 

38 

7 

38 

36.641 

1 

2.359 

39 

8 

29 

29.027 

2 

1.973 

31 

9 

33 

33.493 

3 

2.507 

36 

10 

17 

16.591 

1 

1.409 

18 
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Classification  Table1 


Observed 

Predicted 

Binary.. .if  more  were  awarded 

Percentage 

Correct 

0 

1 

Step  1  Binary.. .if  more  0 

331 

0 

100.0 

Wclc  dWdlUcU 

1 

19 

0 

.0 

Overall  Percentage 

94.6 

a.  The  cut  value  is  .500 


Variables  in  the  Equation 


B 

S.E. 

Wald 

df 

Sig. 

Exp(B) 

95%  C.I.for 

EXP(B) 

Lower 

Upper 

Step  FEAR 
la 

.029 

.031 

.857 

1 

.355 

1.029 

.968 

1.094 

Constant 

-3.445 

.699 

24.276 

1 

.000 

.032 

a.  Variable(s)  entered  on  step  1:  FEAR. 

Regression  FEAR  to  CO  AUTH 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

FEARb 

Enter 

a.  Dependent  Variable:  Contracting  officer  Authority 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.090a 

.008 

.005 

6.005 

a.  Predictors:  (Constant),  FEAR 


AN  OVA3 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Siq. 

1  Regression 

102.573 

1 

102.573 

2.844 

.093b 

Residual 

12550.696 

348 

36.065 

Total 

12653.269 

349 

a.  Dependent  Variable:  Contracting  officer  Authority 

b.  Predictors:  (Constant),  FEAR 


Coefficients3 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Siq. 

B 

Std.  Error 

Beta 

1  (Constant) 

26.419 

.875 

30.190 

.000 

FEAR 

-.070 

.041 

-.090 

-1.686 

.093 

a.  Dependent  Variable:  Contracting  officer  Authority 


Regression  FEAR  Hi/Lo  to  CO  AUTH 

Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

HIGH_FEARb 

Enter 

a.  Dependent  Variable:  Contracting  officer  Authority 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.006a 

.000 

-.003 

6.030 

a.  Predictors:  (Constant),  HIGH_FEAR 


ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

.424 

1 

.424 

.012 

,914b 

Residual 

12652.844 

348 

36.359 

Total 

12653.269 

349 

a.  Dependent  Variable:  Contracting  officer  Authority 

b.  Predictors:  (Constant),  HIGH  FEAR 


Coefficients3 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

25.081 

.458 

54.709 

.000 

HIGH_FEAR 

-.070 

.645 

-.006 

-.108 

.914 

a.  Dependent  Variable:  Contracting  officer  Authority 


Regression  TEE/SSMA  to  CP 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Source  Selection 
Method 

Appropriateness, 
Tech  Eval 
Effectiveness15 

Enter 

a.  Dependent  Variable:  Contractor  Performance 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.350a 

.123 

.118 

7.671 

a.  Predictors:  (Constant),  Source  Selection  Method  Appropriateness,  Tech  Eval 
Effectiveness 


ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

2854.174 

2 

1427.087 

24.255 

,000b 

Residual 

20416.423 

347 

58.837 

Total 

23270.597 

349 

a.  Dependent  Variable:  Contractor  Performance 

b.  Predictors:  (Constant),  Source  Selection  Method  Appropriateness,  Tech  Eval  Effectiveness 


Coefficients3 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

17.765 

2.245 

7.912 

.000 

Tech  Eval 
Effectiveness 

.327 

.102 

.167 

3.209 

.001 

Source  Selection 
Method 

Appropriateness 

.328 

.064 

.268 

5.140 

.000 

a.  Dependent  Variable:  Contractor  Performance 
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Regression  LPTA  Appr  to  CP 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

LPTA 

Appropriatenes 
s  (4  or  less) 
when  LPTA 
usedb 

Enter 

a.  Dependent  Variable:  CP 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

,222a 

.049 

.042 

7.84804 

a.  Predictors:  (Constant),  LPTA  Appropriateness  (4  or  less)  when  LPTA  used 


ANOVA3 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

419.203 

1 

419.203 

6.806 

,010D 

Residual 

8068.526 

131 

61.592 

Total 

8487.729 

132 

a.  Dependent  Variable:  CP 

b.  Predictors:  (Constant),  LPTA  Appropriateness  (4  or  less)  when  LPTA  used 
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Coefficients 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

33.819 

.766 

44.156 

.000 

LPTA 

Appropriateness 
(4  or  less)  when 
LPTA  used 

-4.355 

1.669 

-.222 

-2.609 

.010 

a.  Dependent  Variable:  CP 


Regression 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

LPTA_Appropri 

atenessb 

Enter 

a.  Dependent  Variable:  CP 

b.  All  requested  variables  entered. 


Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

,238a 

.057 

.049 

7.81857 

a.  Predictors:  (Constant),  LPTA_Appropriateness 
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ANOVAa 


Model 

Sum  of 
Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

479.698 

1 

479.698 

7.847 

.006b 

Residual 

8008.032 

131 

61.130 

Total 

8487.729 

132 

a.  Dependent  Variable:  CP 

b.  Predictors:  (Constant),  LPTA_Appropriateness 


Coefficients3 


Model 

Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

25.794 

2.626 

9.821 

.000 

LPTA_Appropriateness 

1.237 

.442 

.238 

2.801 

.006 

a.  Dependent  Variable:  CP 

Regression  CP  to  BS 


Variables  Entered/Removed3 


Model 

Variables 

Entered 

Variables 

Removed 

Method 

1 

Contractor 

Performance13 

Enter 

a.  Dependent  Variable:  BS 

b.  All  requested  variables  entered. 
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Model  Summary 


Model 

R 

R  Square 

Adjusted  R 
Square 

Std.  Error  of 

the  Estimate 

1 

.665a 

.442 

.441 

2.61931 

a.  Predictors:  (Constant),  Contractor  Performance 


ANOVAa 


Model 

Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

1893.312 

1 

1893.312 

275.962 

,000b 

Residual 

2387.546 

348 

6.861 

Total 

4280.857 

349 

a.  Dependent  Variable:  BS 

b.  Predictors:  (Constant),  Contractor  Performance 


Coefficients3 


Model 

Unstandardized  Coefficients 

Standardized 

Coefficients 

t 

Sig. 

B 

Std.  Error 

Beta 

1  (Constant) 

6.506 

.585 

11.121 

.000 

Contractor 

Performance 

.285 

.017 

.665 

16.612 

.000 

a.  Dependent  Variable:  BS 
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